JP2012119804A - Image recorder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an appropriate monitor image which does not damage convenience of a super-wide-angle camera and to facilitate retrieval of a desired subject range from a recorded super-wide-angle image.SOLUTION: An image reader is provided with: imaging means for imaging a super-wide-angle image; recording means for recording data of the captured image; detection means for detecting change in the posture of the imaging means; image cutout means for cutting out a part of the captured image; cutout image change means for changing a position or area of the cutout image in accordance with the detection result by the detection means; and display means for displaying the cutout image. Information on the detection result is added to the image data and recorded. Alternatively, information on the position or area of the cutout image is added to the image data and recorded.

Description

  The present invention relates to an image recording apparatus or an image recording / reproducing apparatus, and more particularly to an image recording or recording / reproducing apparatus having an ultra-wide-angle imaging system, and an image recording method and an image reproducing method thereof.

  Imaging devices such as digital cameras and video cameras use imaging elements such as CCDs and CMOSs as means for converting optical information into electronic data. In addition, the imaging optical system is configured so that the user can freely change the shooting angle of view by the zooming mechanism so that it can cope with various shooting scenes.

  Recently, an increase in the number of pixels of an image sensor has progressed, and an image sensor having a number of pixels sufficiently larger than the number of pixels that can be reproduced by a monitor device such as a television monitor can be obtained. Thereby, a so-called electronic zoom and an electronic camera shake correction function are realized.

  An example of a conventional video camera will be described with reference to FIGS.

  FIG. 13 schematically shows a state in which the conventional video camera 1301 is photographed. A shooting angle of view 1302 is set by the zoom mechanism, and a subject 1303 within the angle of view can be shot. A hand-held image pickup device such as a video camera is difficult to keep stationary during shooting, and the shot image may be blurred as the camera body moves. This is so-called camera shake. A function for correcting the camera shake so that a stable video can be recorded is a camera shake correction. One of the camera shake correction functions is so-called optical camera shake correction in which a part of optical elements such as a lens constituting a zoom mechanism is physically moved to cancel the shake. Alternatively, there is an electronic camera shake correction described below. FIG. 14 is a block diagram showing a configuration related to an electronic camera shake correction function of a video camera. FIG. 15 is a diagram schematically illustrating an imaging region on the imaging element 1401.

  Electronic camera shake correction uses an image sensor 1401 that is larger than the number of pixels necessary to generate a video signal, and from a total image area acquired by the image sensor 1401, captures a predetermined range according to blur information described later. Data is extracted (also referred to as clipping) to generate a video signal.

  As shown in FIG. 15, the entire imaging area 1503 of the imaging device 1401 is larger than the imaging area 1501 necessary for generating a predetermined video signal. A range 1501 indicated by a black broken line is a range necessary for generating a video signal. Normally, image data of the black broken line range 1501 is extracted to generate a video signal. When a camera shake is detected, an imaging region to be extracted to generate a video signal is moved in a direction to cancel the camera shake. A gray broken line 1502 in FIG. 15 represents an imaging region that has moved in a direction to cancel the blur.

  There are typically two methods for detecting blur in electronic camera shake.

  FIG. 14A illustrates a configuration in which a change in posture due to camera shake of the video camera 1301 is detected using gyro sensors 1402 and 1403. Generally, changes in the yaw angle and pitch angle of a video camera are detected by gyro sensors, respectively. In accordance with the detection results of the gyro sensors 1402 and 1403, the control unit 1404 calculates the amount of movement of the extraction area necessary for canceling the change in posture of the video camera 1301. The signal processing circuit 1405 extracts an imaging region and converts it into a predetermined video signal.

  Alternatively, FIG. 14B illustrates a configuration in which movement of a subject image is detected using imaging data. The current image data and the immediately preceding image data stored in the frame memory or the like are compared and evaluated by the motion vector detector 1407, and the moving direction and moving amount of the feature point in the imaging area are calculated from the comparison data. . In the signal processing circuit 1406, the extraction region is moved by the direction and amount that cancel the motion vector.

  With the above-described configuration, the video signal transmitted from the signal processing circuits 1405 and 1406 to the recording / reproducing circuit 1408 becomes a stable video (FIG. 16) with the blur component canceled, and is recorded in the recording memory 1409.

  FIG. 17 is a flowchart for explaining the operation of the video camera 1301 described above.

  FIG. 17A is an operation flow at the time of shooting. In accordance with the operation of the recording button by the user, imaging data is captured from the imaging device to the signal processing circuit (Step 1701). The imaging region is extracted in accordance with the camera shake information by the camera shake detection means described above (Step 1702). As a result, the camera shake component is removed and a stable video is recorded.

  FIG. 17B is an operation flow when the recorded video is reproduced. The recorded video data is read in response to the user operating the playback button (Step 1703). Since the recorded video data is a video from which the camera shake component has already been removed, the recorded video data can be reproduced and viewed as it is on the monitor device.

  Now, various images can be generated in real time from image data obtained by an image sensor by increasing the speed of an image processing IC, improving an image generation algorithm, and the like. In addition, since the high-resolution image is realized by increasing the number of pixels of the image sensor, it is possible to maintain the viewing image quality even when a part of the image is enlarged. For example, it has been commercialized as a function such as electronic zoom and zoom during playback.

  As described above, a sufficiently high-definition image can be obtained, and a desired image can be obtained by real-time processing or post-processing of imaging data, and a new form of electronic camera is proposed. Has been. As an example, there is a camera that captures a super wide-angle image by a fisheye lens or the like with high definition. Since a desired image is obtained by post-processing using a part of the obtained image data, there is no need to manipulate the shooting angle of view, and therefore no zooming mechanism is required, and miniaturization and cost reduction of the device are expected. it can. Furthermore, since the situation around the camera can be recorded at once with an ultra-wide-angle image, it is possible to shoot as long as the subject is around the camera, and it is only necessary to enlarge the subject image to be viewed by the operation during playback. Compared to the conventional case where the camera is always pointed at the subject and the subject's movement is watched, the burden on the photographer is reduced and convenience is improved.

  For example, Patent Document 1 and Patent Document 2 have been proposed as camera devices that can capture a wide-angle image from a conventional camera and obtain a desired image by post-processing.

  Patent Document 1 discloses a television interphone provided with a wide-angle camera. When displaying the visitor's image on the monitor TV, the display position of the visitor on the screen is designated. Thereafter, the display range is changed following the movement of the visitor so that the visitor is always displayed at the designated position.

  Patent Document 2 discloses a television camera device including a fish-eye or other ultra-wide-angle camera. A microphone is provided, the sound source direction is determined, the sound source direction is tracked, and a screen of the sound source direction is extracted to generate a video signal.

  In either case, the display range of the wide-angle image is changed according to external information in a state where the camera posture is fixed.

Japanese Patent Laid-Open No. 6-38214 Japanese Patent Laid-Open No. 11-331827

  Now, when a mobile device such as a video camera is equipped with an ultra-wide-angle camera, it is necessary to devise a method for displaying a monitor image during photographing. Mobile devices are required to be small and light, and thus there are restrictions on the size of a display device such as a liquid crystal display (LCD). Since the entire image captured by the super wide-angle camera is an image having a very wide angle of view including the main subject, it is difficult to confirm the state of the desired main subject from the entire image on the small LCD. Therefore, as mentioned in the reference literature, it is conceivable to cut out only a predetermined range from the wide-angle image and display it on the monitor. However, in a mobile device such as a video camera, since the posture of the device itself can always change, and the above-described camera shake occurs, the conditions for determining the display range must be taken into consideration. Appropriately generate monitor images during shooting so as not to impair the convenience of paying attention to the camera posture during shooting in order to obtain a desired image by post-processing, which is a feature of the ultra-wide-angle camera described above. Must.

  In addition, when reproducing and viewing an ultra-wide-angle image, a desired subject image can be searched and enlarged from the entire image, and the display range can be moved following the movement of the subject image in the entire image. is necessary. In an ultra-wide-angle image taken with a camera of a mobile device, there is a change in posture of the camera itself and camera shake in addition to the movement of the subject itself, and the main subject moves in a complex manner in the entire image. It is necessary to devise in order to stably view the subject image during reproduction.

  In view of the above, an object of the present invention is to provide an easy-to-use imaging apparatus that generates an appropriate monitor image without impairing the convenience of an ultra-wide-angle camera.

  Another object of the present invention is to make it easier to search a desired subject range from recorded super-wide-angle images, and to improve convenience during playback viewing.

The image recording apparatus of the present invention comprises:
An imaging unit that captures a super-wide-angle image, a detection unit that detects a change in the attitude of the imaging unit, a monitor-image clipping unit that clips a part of the super-wide-angle image, and the monitor image clipping position or region according to the detection result A cut-out image changing unit for changing, and a display unit for displaying the cut-out image are provided.
In addition, an image recording means for recording the super wide-angle image is provided.
Alternatively, a storage unit that stores distortion information of an optical element included in the imaging unit is provided, and an image recording unit that records an image generated from the super-wide-angle image according to the distortion information.
The detection information is added to the image data and recorded.
Alternatively, the image cutout position or region information is added to the image data and recorded.
Further, the entire image data is processed according to the read information added to the image data to generate a reproduced image. Then, a display means for displaying the reproduced image or an output means for outputting to the external display means is provided.
Alternatively, a reproduction image is generated by cutting out a predetermined range or region from the image data according to the read information. Then, a display means for displaying the reproduced image or an output means for outputting to the external display means is provided.

  Furthermore, a storage unit that stores distortion information of an optical element included in the imaging unit is provided, and the image data is processed according to the distortion information and the information to generate a reproduced image.

As described above, according to the image recording apparatus of the present invention,
An ultra wide-angle image is captured by the imaging means, a change in the attitude of the imaging means is detected, and a part of the ultra-wide-angle image is changed in position or area according to the detection result, and an image is generated and displayed. Display on means.

  Further, the super wide-angle image is recorded. Alternatively, an image generated from the super-wide-angle image is recorded according to the distortion information of the optical element included in the stored imaging means.

  Further, posture change information of the image pickup means is added to the image data and recorded. Alternatively, the information of the image cutout position or region changed by the posture change information of the imaging unit is added to the image data and recorded.

  Further, the information added to the image data is read, and the entire image data is processed according to the information to generate a reproduced image. Alternatively, a reproduction image is generated by cutting out a predetermined range or region from the recorded image data. The generated reproduced image is displayed on the display means or output to the external display means. Further, distortion information of an optical element included in the imaging unit is stored, and a reproduced image is generated according to the distortion aberration information and the information.

It is an external view of the video camera provided with the fisheye lens which is 1st embodiment. It is the figure which equipped the video camera of 1st embodiment. It is a block diagram of the video camera of a first embodiment. It is a figure showing the whole picture which can be photoed with the video camera of a first embodiment. The state of the movement of the monitor image area in the video camera of the first embodiment is shown. It is a flowchart explaining operation | movement of the video camera of 1st embodiment. It is an external view of the video camera provided with the omnidirectional camera which is 2nd embodiment. It is a block diagram of the video camera of a second embodiment. It is a figure showing the omnidirectional image which can be image | photographed with the video camera of 2nd embodiment. The state of the movement of the monitor image area in the video camera of the second embodiment is shown. The state of the movement of the reproduction | regeneration image area | region in the video camera of 2nd embodiment is represented. It is a flowchart explaining operation | movement of the video camera of 2nd embodiment. It is a figure showing the mode at the time of imaging | photography with the conventional video camera. It is a block diagram explaining the electronic camera-shake correction function in the conventional video camera. It is a figure showing the imaging area on the image pick-up element in the conventional video camera. It is an example of the image to which the camera shake correction in the conventional video camera was applied. It is a flowchart regarding the camera shake correction function of a conventional video camera.

  Hereinafter, embodiments of the present invention will be described.

  In the first embodiment, a method for generating a monitor image during shooting and a method for generating a reproduction image in an ultra-wide-angle camera using a fisheye lens will be described.

  In the second embodiment, a method for generating a monitor image during shooting and a method for generating a reproduction image in a camera (omnidirectional camera) capable of shooting an omnidirectional image of the camera using a mirror will be described.

[Example 1]
FIG. 1 shows the appearance of the video camera 101 according to the present embodiment, and FIG. 2 shows a state when the video camera 101 is used. FIG. 3 is a block diagram showing the configuration of the video camera 101.

  The video camera 101 is a video camera that can capture a super-wide-angle image of approximately 180 ° using a fisheye lens 301, and the camera unit 102 is separated from the recording unit 103 and connected by a transmission cable 104. Further, as shown in FIG. 2, the camera unit 102 is mounted on the photographer's head by the headband 105 and used. As described above, the ultra-wide-angle camera has little need to be aware of the shooting direction and the angle of view. Therefore, if it is worn on the head without being held by hand, the entire range of approximately 180 ° in the front direction of the face of the photographer can be photographed, including the subject noted by the photographer and the surrounding situation. it can.

  The recording unit 103 includes an LCD monitor 302 as a display device. A monitor image, which will be described later, is displayed on the LCD monitor 302 from the entire super wide-angle image captured by the camera unit 102, and the subject image can be confirmed. The recording unit 103 includes a power switch 303 for turning on / off the power of the video camera 101, a recording button 304 for operating recording start / stop, a playback button 305 for reproducing recorded images, and a monitor image to be described later. The direction lever 306 for operating the cutout range is provided.

  FIG. 4 is a schematic diagram for explaining an image that can be captured by the video camera 101 and its subsequent processing.

  FIG. 5 shows the cutout range of the monitor image in the entire image.

  FIG. 6 is a flowchart for explaining the operation of the video camera 101.

  The operation of the video camera 101 will be described with reference to the above.

  FIG. 4 schematically shows an entire image that can be taken by the video camera 101. Since the fisheye lens captures the subject image as a spherical surface, an image in which the real image is distorted toward the periphery as shown in FIG. 4A is obtained. Note that the broken lines shown in FIG. 4A schematically represent how the vertical and horizontal lines of the real image are projected. The image thus obtained is projected onto the image sensor 307 and transmitted to the recording unit 103 via the signal processing circuit 308. The image processing unit 309 provided in the recording unit 103 performs an operation with reference to the stored distortion information of the fisheye lens 301, corrects the distortion as shown in FIG. 401 is obtained. For example, the area of the broken-line circle 402 shown in FIG. 4A is at the center of the shooting angle of view and has little distortion, so the amount of distortion correction is small. On the other hand, the area of the broken-line circle 403 is around the shooting angle of view and is strongly distorted, and the corrected whole image 401 is corrected to an image expanded toward the periphery as indicated by a broken-line circle 404.

  The distortion-corrected whole image 401 is recorded on a memory card 311 as a recording medium via the memory control unit 310 in accordance with an operation of the recording button 304. Further, the monitor image processing unit 312 cuts out a monitor image to be displayed on the LCD monitor 302 from the entire image 401 subjected to distortion correction. Character information (such as operation mode information of the video camera) generated by the character generator 313 is superimposed on the monitor image and displayed on the LCD monitor 302.

  The video camera 101 includes gyro sensors 314 and 315 that detect the posture of the camera unit 102. The gyro sensors 314 and 315 respectively detect changes in the yaw angle and pitch angle of the camera unit 102, calculate the detection results by the calculator 316, and a control unit 317 provided in the recording unit 103 as posture change information of the camera unit 102. Transmit to. The posture change information transmitted from the camera unit 102 is difference information detected at a predetermined interval. The transmission of the posture change information is started from the time when the power is turned on by operating the power switch 303 (FIG. 6, Step 601). By integrating the posture change information in the control unit 317, it is possible to know the current posture of the camera unit 102 with reference to the time when the power is turned on.

  The movement component of the camera unit 102 includes a relatively slow low-frequency component having a large amplitude due to the movement of the photographer's head and the like, and a fast vibration with a small amplitude such as the camera shake described in the background art. There are high frequency components. In the present embodiment, the calculator 316 removes the high-frequency component from the detected posture change information of the camera unit 102 in order to simplify the calculation of the cutout position of the monitor image described later. Accordingly, only the low frequency component of the posture change information of the camera unit 102, that is, relatively large movement information such as when the photographer changes the direction of the head, is transmitted to the control unit 317. The separation between the high frequency component and the low frequency component may be appropriately determined according to the vibration transmission characteristic to the camera unit 102 or the like. For example, the vibration characteristics transmitted to the camera unit 102 differ between when the camera unit 102 is attached to the headband 105 via a damper member such as rubber and when attached to the rigid.

  With the above configuration, how the posture of the camera unit 102 changes is monitored. Corresponding to the posture change of the camera unit 102, the control unit 317 calculates the moving direction and moving amount of the cutout position of the monitor image so as to cancel the posture change of the camera unit 102, and sends the monitor image to the monitor image processing unit 312. Sends out the cut-out position coordinates. As described above, since the entire image 401 is obtained by correcting an image with strong peripheral distortion obtained by the fisheye lens 301, the movement amount of the camera unit 102, the movement amount of the subject image in the entire image 401, and the like. Are not in a fixed relationship. For this reason, the amount of movement of the camera unit 102 is converted into the amount of movement in the entire image 401 subjected to distortion correction, by calculation referring to the distortion aberration information of the fisheye lens 301.

  FIG. 5 shows how the monitor image moves in the entire image 401. In FIG. 5A, a lower left region 501 in the entire image 401 is cut out as a monitor image. At this time, if the photographer wearing the camera unit 102 turns his face in the lower left direction, the subject image originally in the area 501 is the upper right area 502 of the whole image 401 as shown in FIG. Will be moved to. Detecting that the optical axis direction of the camera unit 102 has changed to the lower left direction, and the position of the monitor image cut out from the entire image 401 is an amount obtained by converting the movement amount of the camera unit 102 into the movement amount in the entire image 401. The camera unit 102 moves in the opposite direction to the moving direction. That is, a range to be cut out as a monitor image is changed from the area 501 to the area 502. Accordingly, the change in the posture of the camera unit 102 can be canceled, and a subject image in a certain direction and a certain range can always be cut out as a monitor image regardless of the posture of the camera unit 102.

  Now, the initial cutout range of the monitor image is set according to the operation of the photographer (FIG. 6, Step 602). When the operator operates the direction lever 306 while looking at the LCD monitor 302, the cutout range is sequentially moved from the entire image 401 corrected for distortion, and thus the image is scrolled on the LCD monitor 302. Enlargement / reduction may be performed by other operation buttons, or the aspect ratio may be changed.

  When the photographer presses the recording button 304 (FIG. 6, Step 603), the control unit 317 records the distortion-corrected whole image 401 on the memory card 311 via the memory control unit 310.

  Even if the photographer moves his / her head during photographing, as described above, the movement of the photographer's head is monitored as the posture of the camera unit 102, and the cut-out position of the monitor image is sequentially changed. Therefore, an image in the same direction as the initial cutout range set by the photographer is displayed on the LCD monitor 302 with the same magnification and aspect ratio (FIG. 6, Step 604).

  Further, during recording, the posture change information of the camera unit 102 is added to the image data in association with the time code of the image.

  Now, as described above, the region where the monitor image is cut out from the entire image 401 is calculated based on information obtained by removing high-frequency components from the posture change of the camera unit 102. Therefore, the monitor image is a state in which the vibration of the high frequency component remains in the posture change of the camera unit 102. Therefore, the monitor image cut out from the entire image 401 is compared and evaluated with the previous monitor image stored in the frame memory 319 by the motion vector detector 318, and the motion vector is detected and fed back to the monitor image processing unit 312. . Based on the motion vector, the monitor image processing unit 312 finely adjusts the monitor image cutout range to cancel out the high-frequency component image fluctuation, and displays it on the LCD monitor 302 as a stable image (Step 605 in FIG. 6).

  Next, reproduction of images recorded on the memory card 311 will be described.

  On the memory card 311, an entire image 401 whose distortion has been corrected is recorded. When the playback button 305 is operated, the entire image 401 is read by the memory control unit 310 (FIG. 6B, Step 606) and sent to the monitor image processing unit 312. The viewer moves the cutout range by operating the direction lever 306 and displays the desired range on the LCD monitor 302 for viewing (FIG. 6B, Step 607). Of course, according to other operations, enlargement / reduction, aspect ratio change, and resolution change may be performed.

  Since the recorded whole image 401 does not remove the movement and shaking of the subject image due to the movement of the camera unit 102 at the time of shooting, the desired subject image corresponding to the movement of the camera unit 102 at the time of shooting. Moves in the entire image 401.

  The control unit 317 refers to the posture change information of the camera unit 102 given to the image data, and uses the desired range set by the viewer as a reference, and the entire image of the subject image generated by the change in the posture of the camera unit 102 at the time of shooting. The reproduction cutout range is sequentially changed so as to cancel the movement in the middle (FIG. 6B, Step 608). This can be explained with reference to FIG. When the area 501 shown in FIG. 5A is the viewer's desired reproduction range, the movement is determined by the camera unit attitude information indicating that the optical axis direction of the camera unit 102 has changed to the lower left direction at the time of shooting. The reproduction cutout range is changed from the area 501 to the area 502 so as to cancel.

  Furthermore, since the camera unit posture information is information obtained by removing the high frequency component from the posture change information of the camera unit 102 in the calculator 316, it is a reproduced image in which the image shake of the high frequency component remains. Therefore, the motion vector is detected via the motion vector detector 318 as in the monitor image at the time of shooting described above. Based on the motion vector, the monitor image processing unit 312 finely adjusts the reproduction cutout range, cancels out the high-frequency component image fluctuation, and displays a stable image on the LCD monitor 302 (FIG. 6B, Step 609). .

  At the time of reproduction, it is possible not to move the cutout range of the reproduced image based on the posture change information of the camera unit 102 according to the intention of the user. Even in such a case, the monitor image processing unit 312 is configured to remove high-frequency component image fluctuations. Further, since the entire image is obtained by correcting an image with strong peripheral distortion, the degree of image shake due to camera shake during shooting varies depending on the position where the image is cut out. Although common to the cutout of the monitor image at the time of shooting and the cutout of the reproduced image at the time of reproduction, high-frequency image shaking may be removed using the detection results of the gyro sensors 314 and 315. However, it is necessary to perform a complex conversion operation based on the gyro output according to the image cutout position. Therefore, the gyro sensors 314 and 315 are used as means for detecting a large movement of the camera unit 102, and for high-frequency image fluctuations such as blurring, correction is performed according to the amount of blurring of the image by processing in units of cutout images. Is what you do.

  With the configuration of the video camera 101 described above, when a part of the super-wide-angle image is cut out and displayed as a monitor image, even if the camera unit 102 moves unexpectedly, the desired monitor range can be followed. For example, in the present video camera 101, there may be a situation in which the photographer turns away from the main subject direction, for example, when the photographer performs an operation on the recording unit 103, or when attention is directed to the foot. Even in such a case, the monitor image on the LCD monitor 302 always displays the image in the direction set in the initial state. By looking at the monitor image, it is possible to confirm the state of the subject in the set direction, and to continue shooting with peace of mind.

  Further, during playback, when a part of a super-wide-angle image is cut out for playback and viewing, an image of a desired area can be stably cut out without requiring an operation by the viewer, and convenience is improved.

[Example 2]
FIG. 7 shows the appearance of the video camera 701 in the present embodiment.

  FIG. 8 is a block diagram showing the configuration of the video camera 701.

  In this video camera 701, a mirror 801 having a substantially conical shape and having a predetermined curvature is arranged to face the image sensor 802, and an image around 360 ° around the video camera 701 (hereinafter referred to as an omnidirectional image). .) Can be photographed.

  In addition, an LCD monitor 803 is provided as a display device. A part of the omnidirectional image is cut out and displayed on the LCD monitor 803 as will be described later. Further, a power switch 804 for turning on / off the power of the video camera 701, a recording button 805 for operating recording start / stop, a playback button 806 for reproducing recorded images, and a monitor image clipping range described later are operated. A direction lever 807 is provided.

  FIG. 9 schematically shows an omnidirectional image that can be taken by the video camera 701. FIG. 9A shows an omnidirectional image 910 projected on the image sensor 802. The image sensor 802 projects an omnidirectional image centered on its own position as an image connected in a donut shape. The angle of view in the vertical direction with respect to the real image is determined by the curvature of the surface of the mirror 801, and the real image is projected on the image sensor 802 with distortion. The broken line shown in FIG. 9A schematically represents how the vertical and horizontal lines of the real image are projected. The vertical lines of the real image are projected radially in the radial direction, and the horizontal lines are projected in a concentric manner in which the density changes according to the curvature of the mirror. In order to reproduce such an image on a general monitor device, it is known to develop it into a rectangular flat image 902 with corrected distortion as shown in FIG. 9B. The omnidirectional image shown in FIG. 9A is an image formed by connecting all the surrounding images around the self-position in a donut shape. Therefore, in order to develop into a rectangular image, it is necessary to cut in some direction. Here, when the omnidirectional image 910 is cut using the azimuth indicated by the arrow 901 as a cutting portion and corrected and developed into a rectangular image, a planar image 902 as shown in FIG. 9B is obtained. In FIG. 9A, the area of the broken line circle 903 at the center of the image is expanded into a rectangular area 904 over the entire width of the bottom of the image in FIG. 9B, and the broken line circle areas 905 and 906 each correct distortion. Thus, the deformed areas 907, 908, and 909 are developed. In particular, a region 906 straddling the cut portion (the direction indicated by the arrow 901) is regions 908 and 909 separated at both ends of the corrected and developed image.

  FIG. 12 is a flowchart for explaining the operation of the video camera 701.

  The operation of the video camera 701 will be described with reference to FIGS.

  The video camera 701 sends the donut-shaped omnidirectional image shown in FIG. 9A projected on the image sensor 802 to the memory control unit 809 via the signal processing circuit 808, and operates the recording button 805 of the photographer. In response, the data is recorded on the memory card 810.

  This video camera 701 is equipped with gyro sensors 811, 812, 813 for detecting a change in posture of this apparatus. The gyro sensors 811, 812, 813 detect changes in the yaw angle and pitch angle of the video camera 701 and the rotation angle around the center axis of the mirror 801, respectively, and the detection results are calculated by the calculator 814 to obtain posture change information. Is transmitted to the control unit 815. The posture change information is difference information detected at a predetermined interval, and transmission of the posture change information is started when the power source switch 804 is operated to turn on the power of the apparatus (FIG. 12, Step 1201). By integrating the posture change information in the control unit 815, it is possible to know the current posture of the video camera 701 with reference to the time when the power is turned on.

  In this embodiment as well, as in the first embodiment, the calculator 814 removes the high frequency component from the detected posture change information of the video camera 701. Accordingly, a low-frequency component resulting from a relatively large movement among the movements of the video camera 701 is transmitted to the control unit 815.

  Furthermore, when the photographer operates the recording button 805, the recording of the omnidirectional image is started (FIG. 12, Step 1203).

  Now, the video camera 701 monitors how the posture of the video camera 701 has changed, and controls the position where the monitor image to be displayed on the LCD monitor 803 is extracted from the omnidirectional image (FIG. 12, Step 1204). Hereinafter, the control of the position where the monitor image is extracted will be described.

  FIG. 10 shows how the monitor image area of the video camera 701 is cut out. As shown in FIG. 10A-1, it is assumed that a person's face vicinity 1001 is designated as the monitor position in the omnidirectional image. As information on where the region extracted for the monitor image is located in the omnidirectional image, the azimuth (the position indicated by the arrow 1004) and the radial position of the center of the cutout range are associated with the time code of the image in the image data. It gives (FIG. 12, Step 1205).

  Now, the monitor image processing unit 816 refers to the stored mirror curvature information, calculates an area in which distortion can be corrected to an appropriate image size corresponding to the aspect ratio of the LCD monitor 803, and FIG. An image of a substantially fan-shaped region 1002 shown is extracted. Distortion correction is performed on the extracted image of the substantially fan-shaped region 1002 to obtain a rectangular monitor image 1003 shown in FIG. 10A-3 (FIG. 12, Step 1206). As described above, since the omnidirectional image is an image in which the density is changed, the shape of the region changes depending on the extraction position even when the distortion is corrected to the same image size.

  The monitor image is displayed on the LCD monitor 803 with the character information generated by the character generator 817 superimposed.

  As described above, the monitor image area extracted from the omnidirectional image is calculated based on information obtained by removing the high frequency component from the posture change of the video camera 701. Therefore, the monitor image is a state in which the high-frequency component still remains in the posture change of the video camera 701. Therefore, the motion vector detector 818 compares and evaluates the previous monitor image stored in the frame memory 819, detects the motion vector, and feeds it back to the monitor image processing unit 816. Based on the motion vector, the monitor image processing unit 816 finely adjusts the monitor image cutout range to cancel out the image shake, and displays the image on the LCD monitor 803 as a stable image (Step 1207 in FIG. 12).

  Now, when the video camera 701 moves during shooting, the position of the subject image changes in the omnidirectional image. Since the video camera 701 monitors the attitude change of the apparatus by the gyro sensors 811, 812, and 813, the monitor image extraction area is moved in a direction to cancel the change corresponding to the detected attitude change. For example, FIG. 10B-1 shows an omnidirectional image when the video camera 701 rotates around the center axis of the mirror 801. The desired subject image in the monitor image extraction area until then has moved to a new position 1005 even in the omnidirectional image due to the change in the attitude of the video camera 701. The monitor image processing unit 816 refers to the posture change information of the video camera 701, and changes the orientation of the region from which the monitor image is extracted and the position in the radial direction so as to cancel the information (FIG. 12, Step 1204).

  Furthermore, as information on where the changed monitor image extraction region is located in the omnidirectional image, the azimuth (indicated by the arrow 1008) and the radial position are associated with the time code of the image and are added to the image data.

  Then, with reference to the stored mirror curvature information, a new extraction area that can be appropriately displayed on the LCD monitor after distortion correction is calculated. As described above, the shape of the extraction region can change depending on the density of the omnidirectional image. Based on the calculation result, an image region 1006 (FIG. 10B-2) is extracted, and distortion correction is performed to generate a monitor image 1007 (FIG. 10B-3) (FIG. 12, Step 1206). In the same manner as described above, the motion vector detection removes high-frequency component image fluctuations and displays them on the LCD monitor 803 (FIG. 12, Step 1207).

  The change of the monitor position and the recording of the position information described above are repeated as long as the attitude of the video camera 701 is changed.

  As described above, the video camera 701 always displays a certain position as a monitor image in a certain azimuth regardless of a change in the posture of the apparatus.

  Note that the initial position of the monitor image may be set by the photographer operating the direction lever 807 as in the first embodiment (FIG. 12, Step 1202). When the image is enlarged or reduced or the image size is set by another operation, the monitor image area to be extracted may be calculated using these as parameters.

  Next, reproduction of images recorded on the memory card 810 will be described.

  An omnidirectional image as shown in FIG. 9A is recorded in the memory card 810. An omnidirectional image is very different from the human field of view, so it is not suitable for playback on a general television monitor. For this reason, as described above, the image is developed into a rectangular planar image with distortion corrected. Further, the flat image after the development is an image obtained by photographing the entire circumference of 360 ° and is a wide span image, and the aspect ratio is greatly different from that of a general monitor device. Therefore, it is realistic to cut out and appreciate a part of the planar image.

  By the operation of the playback button 806, the omnidirectional image is read out by the memory control unit 809 and sent to the image processing unit 820 (FIG. 12, Step 1208). As described with reference to FIG. 9, the image processing unit 820 cuts the donut-shaped omnidirectional image at some direction, performs distortion correction, and develops it into a rectangular planar image (FIG. 12, Step 1209).

  Here, the orientation information for cutting and developing the donut-shaped omnidirectional image uses the orientation information of the monitor position at the time of shooting given to the image data.

  FIG. 11 shows a state of correction development of an omnidirectional image during image reproduction. The orientation information of the monitor extraction area displayed on the LCD monitor 803 at the time of photographing is given to the image data, and the orientation information changes every moment according to the movement of the video camera 701 at the time of photographing. FIGS. 11A and 11B show examples in which the orientation information of the monitor position given to the image data changes. When the video camera 701 rotates around the central axis of the mirror 801 during shooting, the orientation of the monitor position is the position indicated by the arrow 1101 in FIG. 11A, but the arrow 1102 in FIG. The direction has changed. The image processing unit 820 reads the orientation information of the monitor extraction area at the time of shooting from the image data, changes the cutting position of the donut-shaped omnidirectional image, and the orientation of the monitor extraction area at the time of shooting always becomes the center of the planar image. The correction is expanded. That is, in the omnidirectional images shown in FIGS. 11A and 11B, as shown in FIG. 11C, the position 1103 of the person as the monitor position at the time of shooting is positioned at the center of the planar image 1104. The correction is expanded.

  After the omnidirectional image is developed into a flat image as described above, the viewer operates the direction lever 807 to change the cutout position so that a desired subject image appears on the monitor screen (FIG. 12, Step 1210). . Specifically, as shown by a broken line in FIG. 11C, when the viewer operates the direction lever 807 while looking at the monitor screen, the reproduction cutout range is sequentially moved, so that the image scrolls on the monitor screen. Go. As a result, the initial reproduction screen is changed from the range 1105 near the face of the person as shown in FIG. 11D to the surrounding landscape range 1106 as shown in FIG. 11E. Of course, enlargement or reduction may be performed by another operation, or the aspect ratio of the screen may be changed.

  In the reproduced image generated as described above, the motion vector is detected by the motion vector detector 818 and fed back to the monitor image processing unit 816 in the same manner as at the time of shooting. Based on the motion vector, the monitor image processing unit 816 finely adjusts the reproduction image cut-out range to remove high-frequency component image fluctuations due to blurring at the time of shooting and stabilize the reproduction image (FIG. 12, Step 1211). ).

  As described above, according to the present video camera 701, when a part of an omnidirectional image is cut out and displayed as a monitor image, even if the video camera 701 moves unexpectedly, the desired monitor position is tracked without being lost. Continue to display the image of the monitor position set to. By looking at the monitor image, it is possible to confirm the state of the set subject, and to continue shooting with peace of mind.

  Also, during playback, when a part of an omnidirectional image is cut out and viewed for viewing, the orientation in the monitor direction at the time of shooting is always the center of the planar image, making it easier to search for a desired subject and increasing convenience. .

101 Video camera 102 Camera unit 103 Recording unit 104 Transmission cable 105 Headband 301 Fisheye lens 302 LCD monitor 303 Power switch 304 Recording button 305 Playback button 306 Direction lever 307 Image sensor 308 Signal processing circuit 309 Image processing unit 310 Memory control unit 311 Memory card 312 Monitor image processing unit 313 Character generator 314 Gyro sensor 315 ... Gyro sensor 316 ... Operation unit 317 ... Control unit 318 ... Motion vector detector 319 ... Frame memory 701 ... Video camera 801 ... Mirror 802 ... Imaging device 803 ... LCD monitor 804 ... Power switch 805 ... Record button 806 ... Play button 807 Direction lever 808 Signal processing circuit 809 Memory control unit 810 Memory card 811 Gyro sensor 812 Gyro sensor 813 Gyro sensor 814 Operation unit 815 Control unit 816 Monitor image Processing unit 817 ... Character generator 818 ... Motion vector detector 819 ... Frame memory 820 ... Image processing part 1301 ... Video camera 1302 ... Shooting angle of view 1303 ... Subject 1401 ... Imaging element 1402 ... Gyro sensor 1403 ... Gyro sensor 1404 ... Control unit 1405 ... Signal processing circuit 1406 ... Signal processing circuit 1407 ... Motion vector detector 1408 ... Recording / playback circuit 1409 ... Recording memory 1501 ... Imaging area 1502 ... Imaging area 1503... All imaging areas of the image sensor

Claims (8)

An imaging means for capturing an ultra-wide-angle image;
Detecting means for detecting a change in posture of the imaging means;
Monitor image cutout means for cutting out a part from the super wide-angle image;
In accordance with the detection result of the detection means, cut-out image change means for changing the monitor image cut-out position or region;
Display means for displaying the clipped image;
An image recording apparatus comprising: The image recording apparatus according to claim 1, further comprising an image recording unit that records the captured ultra-wide angle image. Storage means for storing distortion information of optical elements included in the imaging means;
Image recording means for recording an image generated from the super-wide-angle image according to the distortion aberration information;
An image recording apparatus according to claim 1 or 2, further comprising: The image recording apparatus according to claim 2, further comprising an image recording unit that records the detection information added to the image data. 5. The image recording apparatus according to claim 2, further comprising an image recording unit configured to record the monitor image cutout position or the region information in addition to the image data. Information reading means for reading the information added to the image data;
Reproduction image generation means for processing the image data and generating a reproduction image according to the information;
Display means for displaying the reproduced image, or output means for outputting to the external display means;
The image recording apparatus according to claim 2, further comprising: Information reading means for reading the information added to the image data;
In accordance with the information, reproduced image generating means for cutting out a predetermined position or region from the image data and generating an image;
Display means for displaying the reproduced image and output means for outputting to the external display means;
The image recording apparatus according to claim 2, further comprising: Storage means for storing distortion information of optical elements included in the imaging means;
Reproduction image generation means for processing the image data and generating a reproduction image according to the distortion aberration information and the information,
An image recording apparatus according to claim 2, comprising: