JP2005252733A - Method, and program for image editing and recording medium for recording image-editing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image-editing device etc. capable of simplifying the structure of a camera by easily performing image-editing, along the intention of a photographer. <P>SOLUTION: In the image-editing device, biological information obtained by a line of sight/angular velocity detector 7, the information, such as the line of sight direction or the angular velocity of a head when a user observes an object is recorded in a recording memory 157 so that the information is correlated to time information from a timer 123 about the time when the biological information is recorded, and image data are recorded in the recording memory 157 so that the data are correlated to the time information. The recorded image data are read, and a first CPU 161 applies predetermined editing processing to the image data, on the basis of the biological information recorded at the same time as the read image data. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image editing apparatus, an image editing method, an image editing program, and a recording medium for recording an image editing program for performing editing processing on image data.

  Conventionally, imaging devices such as video cameras and digital cameras have been widely used for the purpose of capturing subject images.

  When taking a picture using such a conventional imaging device, the photographer must hold the camera and perform an operation such as zooming while checking the subject through the camera finder. For this reason, the photographer has to concentrate on the photographing operation, and even if the photographing scene is various events such as athletic meet or festival, the photographer himself cannot participate in these events. In particular, since a video camera captures images at continuous time intervals, it is more difficult to participate in an event than a digital camera that captures still images.

  Conventionally, there has been known an imaging apparatus that shoots while wearing the head. As such a technique, for example, in Japanese Patent Laid-Open No. 2001-281520 (Patent Document 1), a line-of-sight detection sensor is provided so as to be arranged near the user's eyeball, and the state of the user's eyeball is detected by the line-of-sight detection sensor. And an optical apparatus configured to perform focus adjustment, zoom control, and lens rotation control based on the line-of-sight direction based on the detection result. In particular, with regard to lens rotation control, the camera itself is rotated based on line-of-sight information so that the shooting direction coincides with the line-of-sight direction. This publication also describes a technique for performing zoom control based on the movement of the crystalline lens of the user's eyeball.

  On the other hand, conventionally, an image editing apparatus that performs a predetermined editing process on a photographed image has been proposed.

For example, Japanese Patent Laid-Open No. 9-130732 (Patent Document 2) describes a moving image editing apparatus configured to be able to cope with different scene changes. That is, the moving image editing apparatus compares each frame image of the input video signal with a frame image adjacent to the frame image, and detects the comparison result as a video change degree. If the detected degree of change exceeds a predetermined threshold value, it is determined that there is a scene change that is a scene change. If it is determined that this scene change exists, based on the continuity of the degree of change, it is determined whether the scene change instantaneously changes scenes or gradually changes scenes. Further determine. Here, when it is determined that the scene change is instantaneously switched, the determined time point is detected as an instantaneous scene change, while the scene change is determined to be gradually switched. In this case, the period from the determined time point to the time point when the change degree of the video is stabilized is detected as a period of gradual scene change. This moving image editing apparatus performs moving image editing corresponding to both the instantaneous scene change and the gradual scene change thus detected.
JP 2001-281520 A JP-A-9-130732

  However, in the configuration described in Patent Document 1, since the camera itself is rotated in order to make the photographing direction coincide with the line-of-sight direction, the rotation mechanism of the camera becomes complicated, large and heavy, It cannot be said that it is suitable for an imaging device that is worn on the head. Further, since it is complicated to adjust the visual axis and the optical axis of the camera, it is difficult to perform accurate control. Furthermore, zoom control based on the movement of the crystalline lens of the eyeball as described in Patent Document 1 is not necessarily a technique that can perform zooming in accordance with the user's intention. In other words, since the focus information (distance information to the subject) is obtained by the movement of the crystalline lens, even if the user looks far away, it does not necessarily lead to the intention to perform the zoom operation. Because.

  Further, the technique described in Patent Document 2 is not necessarily image editing based on the photographer's intention, and is not necessarily image editing based on the photographer's movement or the like. It was not always possible to obtain editing results. Furthermore, the technique described in Patent Document 2 has a problem that it takes a long time for the editing process because the degree of change is detected for all frame images.

  The present invention has been made in view of the above circumstances, and provides an image editing apparatus, an image editing method, an image editing program, and a recording medium for recording the image editing program, which can simplify the configuration of the camera. It is an object.

  Another object of the present invention is to provide an image editing apparatus, an image editing method, an image editing program, and a recording medium for recording the image editing program, which can easily perform image editing in accordance with a photographer's intention. Yes.

  In order to achieve the above object, the image editing apparatus according to the first invention records biological information related to the movement of the photographer when observing the subject in association with time information related to the time when the biological information was recorded. Biometric information recording means, image data recording means for recording image data in association with the time information, and image data recorded by the image data recording means are read and recorded at the same time as the read image data. And processing means for performing a predetermined editing process on the image data based on the biometric information.

  An image editing apparatus according to a second invention is the image editing apparatus according to the first invention, wherein the image data is obtained by photographing with a camera mounted on a photographer's head. .

  Furthermore, in the image editing device according to the third invention, in the image editing device according to the first invention, the biological information is angular velocity information related to an angular velocity of a photographer's head.

  An image editing apparatus according to a fourth invention is the image editing apparatus according to the third invention, wherein cutting, blurring correction, extending recording time, and electronic zooming of recorded image data are performed based on the angular velocity information. And at least one of the above-described editing processes.

  An image editing apparatus according to a fifth invention is the image editing apparatus according to the first invention, wherein the biological information is gaze direction information related to a gaze direction of the photographer.

  An image editing apparatus according to a sixth invention is the image editing apparatus according to the fifth invention, wherein electronic zooming of recorded image data is performed as the editing process based on the line-of-sight direction information. It is.

  An image editing apparatus according to a seventh invention is the image editing apparatus according to the fifth invention, wherein the line-of-sight direction information includes at least one of a movement speed in the line-of-sight direction and a continuous movement amount in the line-of-sight direction. It is.

  An image editing apparatus according to an eighth aspect of the invention is the image editing apparatus according to the first aspect of the invention, wherein information relating to editing processing can be corrected by manual operation.

  According to a ninth aspect of the invention, there is provided an image editing method for recording biometric information and image data related to a movement of a photographer when observing a subject in association with time information related to the time when the biometric information and the image data were recorded; And reading the recorded image data and performing a predetermined editing process on the image data based on the biometric information recorded at the same time as the read image data.

  An image editing program according to a tenth aspect of the invention records the biometric information and the image data related to the movement of the photographer when observing the subject in association with the time information related to the time when the biometric information and the image data were recorded; A program for causing the computer to execute a predetermined editing process on the image data based on the biometric information recorded at the same time as the read image data and the read image data. .

  According to an eleventh aspect of the present invention, there is provided a recording medium for recording an image editing program, associating biometric information and image data related to a photographer's movement when observing a subject with time information relating to the time when the biometric information and image data are recorded And performing a predetermined editing process on the image data based on the biometric information recorded at the same time as the read image data and reading the recorded image data. This is a recording medium for recording an image editing program.

  According to the image editing apparatus, the image editing method, the image editing program, and the recording medium for recording the image editing program of the present invention, the configuration of the camera can be simplified. In addition, it is possible to easily perform image editing in accordance with the photographer's intention.

  Embodiments of the present invention will be described below with reference to the drawings.

  1 to 70 show Embodiment 1 of the present invention, and FIG. 1 is a perspective view showing a usage form of a head-mounted camera.

  Note that the head-mounted camera of the present embodiment is an imaging device if attention is focused on the function of capturing an image, and is also an image playback device if attention is focused on the function of reproducing an image. As will be described later, a head-mounted display device with a risk prevention function is provided. Furthermore, since this head-mounted camera can also edit an image, it also serves as an image editing device.

  The image reproducing apparatus can also be applied as an image reproducing method and an image reproducing program. By recording the image reproducing program on a recording medium, a recording medium for recording the image reproducing program is obtained.

  Similarly, the image editing apparatus can also be applied as an image editing method or an image editing program. By recording the image editing program on a recording medium, a recording medium for recording the image editing program is obtained.

  As shown in FIG. 1, the head-mounted camera (hereinafter abbreviated as “camera” as appropriate) 1 includes a head-mounted unit 2 having a substantially glasses shape, and a connection with the head-mounted unit 2, for example. A control / recording unit 4 as a main unit connected via a cable 3 as a means, a remote control unit (hereinafter abbreviated as a remote control unit) 5 for remotely performing an operation input related to the camera 1, and It is divided roughly into.

  The head-mounted unit 2 is configured to be able to substantially directly observe a subject that is an observation target during see-through display and also to capture the subject. The head mounting part 2 is mounted on the head and used in the same manner as general glasses for diopter correction, as can be seen from the shape of the glasses. Etc. are also reduced in size and weight so as to approximate to normal glasses as much as possible.

  The cable 3 connects the connection terminal 3a provided on one end side to the cable connection terminal 32 (see FIG. 2 and the like) of the head mounting portion 2 and the connection terminal 3b provided on the other end side of the control / recording unit. The head mounting unit 2 and the control / recording unit 4 are connected to each other by connecting to the cable connection terminal 49 (see FIG. 6). Here, as the connection means for electrically connecting the head mounting unit 2 and the control / recording unit 4, a wired cable 3 is used. However, the present invention is not limited to this, and for example, means for communicating with each other wirelessly May be used.

  The control / recording unit 4 includes image data recording means, biological information recording means, processing means, parallax correction means, calculation means, danger prevention means, relative speed detection means, relative speed distance detection means, gait detection means, and corneal reflection. It also serves as a light detection means, a region setting means, a minimum value detection means, a pupil detection means, and a line-of-sight direction calculation means, and controls the entire camera 1 and also controls the image captured by the head mounting unit 2. It is configured to function as an image editing apparatus for recording and further editing the recorded image. The control / recording unit 4 can be used in various states such as a state where it is attached to a waist belt or the like, or a state where it is stored in an inner pocket of a jacket, etc. The size and weight are reduced. Of course, it is also possible to use the control / recording unit 4 in a state of being accommodated in a bag or the like by using a long cable 3 or the like.

  The remote controller 5 is for a photographer to perform relatively frequently operations such as see-through display control and imaging operation control of the head-mounted unit 2 by remote operation. Accordingly, the remote control unit 5 is configured to be small and light enough to fit in the palm of one hand, for example, and communicate with the control / recording unit 4 wirelessly, for example. It has become.

  The head mounting unit 2, the control / recording unit 4, and the remote control unit 5 are configured separately from each other in the present embodiment, and thereby the mounting feeling by reducing the size and weight of the head mounting unit 2 is achieved. Improvement of operability by adopting the remote control unit 5 and the like.

  Next, with reference to FIGS. 2 to 4, the appearance and outline of the head mounting portion 2 will be described. 2 is a front view showing the head mounting portion, FIG. 3 is a plan view showing the head mounting portion, and FIG. 4 is a right side view showing the head mounting portion.

  The head mounting portion 2 is a front portion 11 that is a portion corresponding to a lens, a rim, a bridge, a wisdom or the like in general eyeglasses, and from the left and right sides of the front portion 11 toward the rear (opposite the subject) Each temple portion 12 extends and can be folded with respect to the front portion 11.

  The front portion 11 includes a frame portion 13 that is a holding unit that incorporates a part of an optical system for image display, a part of an optical system for detecting a visual line direction, an electric circuit, and the like.

  The frame unit 13 has a light emitting unit 16 for light projection, which is a distance measuring unit used for measuring the distance to the subject in a substantially central part, and a first unit for recording the sound from the subject side in stereo on both the left and right sides. A first microphone 17 and a second microphone 18 are provided.

  Further, transparent optical members 14 and 15 as light guide members are attached to the frame portion 13 so as to correspond to the left and right eyes, respectively. One of the transparent optical members 14 and 15 is used to detect the line-of-sight direction as described later, and the other transparent optical member 15 displays an image as described later. Is used for this purpose. And these transparent optical members 14 and 15 are formed so that it may become the minimum magnitude | size required in order to fulfill | perform each function.

  The transparent optical member 14 is provided with a HOE (Holographic Optical Element) 24, a half mirror 128, and a HOE 129. The transparent optical member 15 is provided with a HOE 25 as a combiner. Yes. Details of each optical system will be described later.

  In addition, lenses 21 and 22 for diopter adjustment are detachably attached to the frame portion 13 so as to be positioned on the subject side of the transparent optical members 14 and 15, respectively.

  That is, a rim 20 as an attaching means for attaching the lenses 21 and 22 is attached to the frame portion 13 by, for example, screws 28 at the center side, the left side of the left eye, and the right side of the right eye, respectively. It can be attached.

  The rim 20 is provided with a pair of nose pads 19 via a pair of clings 23 for placing the head mounting portion 2 on the nose bridge by supporting the nose bridge from the left and right.

  In such a configuration, the rim 20 and the lenses 21 and 22 can be easily removed by removing the screw 28, and the lenses 21 and 22 can be adapted to other diopters. It is also easy to replace and reinstall.

  At this time, if the rim 20 is formed of a material having elasticity of a predetermined level or more, only one of the left-eye lens 21 and the right-eye lens 22 can be selectively selected by removing or loosening one screw 28. (Independently) can be attached and detached to increase convenience.

  As described above, this head-mounted camera is a head-mounted camera with a diopter adjusting lens. If attention is paid to the function of displaying an image, the head-mounted display device with a diopter adjusting lens is It has also become a thing.

  The diopter adjustment lenses 21 and 22 can be lenses that are used in general glasses, and can be configured at low cost.

  Further, in the above description, the diopter adjusting lenses 21 and 22 are disposed on the subject side of the transparent optical members 14 and 15, but not limited thereto, on the back side (eye side) of the transparent optical members 14 and 15. It is of course possible to arrange them.

  Furthermore, an image pickup unit 30 that is an image pickup unit for picking up a subject image and also serves as a distance measurement unit is provided at the joint 29 on the side surface of the left side of the frame unit 13 (that is, the right side in FIGS. 2 and 3). Via the pedestal 33, the photographing direction is fixed so as to be adjustable.

  The pedestal 33 is attached to the joint 29 by screws 34 and 35, and is configured to attach the imaging unit 30 on itself through screws 36 and 37. By adjusting the relative angle of the imaging unit 30 with respect to the front unit 11 via the pedestal 33, the imaging optical system 31 (see also FIG. 11 and the like) included in the imaging unit 30 is adjusted as will be described later. The optical axis and the visual axis can be adjusted. Adjustment of the mounting position of the imaging unit 30 in such a configuration will be described later with reference to FIGS. 19 and 20.

  A cable connection terminal 32 for connecting the connection terminal 3 a on one end side of the cable 3 is provided on the lower end side of the imaging unit 30.

  A cable 38 is once extended from the back side of the imaging unit 30 and is connected to the frame unit 13 after diving under the temple unit 12 on the left eye side. Thereby, the electric circuit in the frame unit 13 and the electric circuit in the imaging unit 30 are connected to each other.

  The temple portion 12 is connected to the front portion 11 using hinges 78 and 79, so that the temple portion 12 can be folded with respect to the front portion 11. That is, when not in use, the temple portion 12 can be bent toward the center portion of the front portion 11, and the folded position can be taken along the front portion 11. It is possible to do. In addition, tip cell moderns 26 to be put on the ears are provided at the tip portions of the left and right temple portions 12, respectively.

  The temple portion 12 on the right eye side is provided with a storage portion 27 for storing various electronic circuits related to the head mounting portion 2 such as angular velocity sensors 141 and 142 (see FIG. 11) described later. . For example, a switch 39 serving as a danger prevention operation unit for forcibly prohibiting see-through display of an image is disposed on the upper surface of the storage unit 27 so that the switch 39 can be operated as desired from the outside. A cable 27 a is extended from the storage portion 27 and connected to each circuit in the frame portion 13. Further, the control / recording portion 4 is connected via a circuit in the imaging portion 30. It is designed to be connected to the circuit.

  Next, the external appearance and outline of the control / recording unit 4 will be described with reference to FIGS. 5 is a plan view showing the control / recording unit with the operation panel closed, FIG. 6 is a right side view showing the control / recording unit with the operation panel closed, and FIG. 7 is a control with the operation panel closed. FIG. 8 is a plan view showing operation switches arranged on the operation panel, and FIG. 9 is a perspective view showing the control / recording unit in a state where the operation panel is opened.

  The control / recording unit 4 includes a control / recording main body 41 and an operation panel 42 that can be opened and closed with respect to the control / recording main body 41 via a hinge 43. .

  The control / recording main body 41 incorporates each circuit as will be described later, and is an LCD display element (hereinafter referred to as a display means) configured as a liquid crystal monitor at a position where observation is possible when the operation panel 42 is opened. , Abbreviated as “LCD”) 48. The LCD 48 is used not only for displaying an image during reproduction, but also for displaying a menu screen for setting various modes. Further, the control / recording main body 41 is formed with a recess 45 so that a fingertip or the like can be easily applied when the operation panel 42 is opened and closed.

  Further, as shown in FIG. 6, a lid 52 that can be opened and closed with respect to the control / recording main body 41 by a hinge 46 is provided on the right side surface of the control / recording main body 41. The locked portion 52a is locked to the locked portion 52b on the control / recording main body 41 side so that the closed state is maintained. When the lid 52 is opened, as shown in FIG. 6, a cable connection terminal 49 adapted to be connected to the cable connection terminal 32 of the head mounting portion 2 via the cable 3 is connected to the television. The AV / S connection terminal 50 that is a terminal of the PC and the PC connection terminal 51 that is a terminal for connecting to a personal computer (PC) are exposed. In this way, the cords are connected together on the right side of the control / recording main body 41, so that the cords do not extend from the other side, Annoyance can be reduced.

  On the other hand, as shown in FIG. 7, a lid 53 that can be opened and closed with respect to the control / recording main body 41 by a hinge 47 is provided on the left side surface of the control / recording main body 41. The closed state is maintained by locking the locking portion 53a to the locked portion 53b on the control / recording main body 41 side. When the lid 53 is opened, as shown in FIG. 7, a recording memory insertion port 54 for inserting a recording memory 157 (see FIG. 11 etc.) as a detachable recording means composed of a card memory or the like, and a power source And a battery insertion port 55 for removably inserting a battery for supplying the battery.

  As shown in FIG. 5, the operation panel 42 is provided with a power switch 44 on the outer side exposed to the outside even in the closed state, and further on the inner side exposed to be operable only in the opened state. Various operation switches as shown in FIG.

  That is, on the inner surface side of the operation panel 42, a speaker 56 for reproducing sound, a switch 57 for increasing the volume of the sound generated from the speaker 56, and a switch 58 for decreasing the volume A playback / stop switch 59 for playing back or pausing the image information recorded in the recording memory 157, a switch 61 for fast-forwarding and searching for an image, and a forward-looking image. A switch 62 for fast-forwarding and searching, a menu button 63 for displaying on the LCD 48 a menu screen for setting various functions and dates related to the camera 1, and the menu screen are displayed on the menu screen. Menu selection for moving the item of interest up, down, left, right in each item and scrolling display information A switch 66, 67, 68, 69, and confirmation switch 65 for determining the focused item or the like displayed are disposed.

  It should be noted that special mode settings that are not normally used, such as settings relating to the gaze direction detection calibration mode described later, are performed by operating the menu button 63 and selecting a predetermined menu. .

  As described above, the switches arranged on the operation panel 42 are mainly switches for setting information with a relatively low frequency of change during the photographing operation.

  Next, the external appearance and outline of the remote control unit 5 will be described with reference to FIG. FIG. 10 is a plan view showing the configuration of the remote control unit.

  As described above, the remote controller 5 is provided with switches for changing information at a relatively high frequency during a photographing operation. As shown in FIG. 10, the FA / A / An M switch 71, an F / V switch 72, a release switch (REL) 73, a recording switch (REC) 74, a zoom switch 75, and an exposure correction switch 76 are included.

  The FA / A / M switch 71 is a switching unit, and performs an operation of enlarging a captured image corresponding to a captured image frame indicating a shooting range and performing a see-through display as an electronic view during telephoto shooting with a predetermined focal length or longer. This is for switching between fully automatic (FA: full auto mode), automatic (A: auto mode) or manual (M: manual mode).

  The F / V switch 72 is switching means, and the see-through display on the transparent optical member 15 is set to a photographic image frame (F) indicating a photographic range, or a photographic image (V) ( Electronic view) or for switching.

  The release switch (REL) 73 is for starting to capture a still image with higher definition than a moving image.

  The recording switch (REC) 74 is for switching each time a moving image recording start and a recording stop are pressed.

  The zoom switch 75 is a photographic image frame setting means, and is a telescope for performing zoom (optical zoom and / or electronic zoom) of the imaging unit 30 including the photographic optical system 31 on the telephoto (T: tele) side. The switch 75a includes a wide switch 75b for performing on the wide angle (W: wide) side.

  The exposure compensation switch 76 includes a minus exposure compensation switch 76a for performing exposure compensation of a photographed image on the minus side, and a plus exposure compensation switch 76b for performing exposure compensation on the plus side. ing.

  Since the zoom operation of the photographic optical system 31 and the change of the viewing angle of the photographic image frame observed from the photographer who is also the observer are configured to be interlocked, the zoom switch 75 is In other words, it can be paraphrased as having a tele switch 75a for reducing the viewing angle of the photographic image frame observed by the photographer and a wide switch 75b for enlarging the viewing angle.

  Further, the “shooting image frame” is an index representing the range of the subject shot by the imaging unit 30 (see FIG. 21 and the like).

  FIG. 11 is a block diagram showing a configuration mainly related to an electronic circuit of the head-mounted camera.

  As described above, the configuration of the camera 1 includes a head mounting unit 2 having an imaging function, a function of recording an image photographed by the head mounting unit 2, and a function of controlling the camera 1 as a whole. Are roughly divided into a control / recording unit 4 provided with a remote control unit 5 for remotely operating the camera 1. Among these, the head-mounted unit 2 further includes an imaging unit 30 that is an imaging unit for performing imaging, and a see-through image display unit 6 that is a display unit for mainly performing see-through display and also serves as a danger prevention unit. Detecting the gaze direction of the photographer or detecting the angular velocity associated with the movement of the head of the photographer, and detecting the gaze direction / angular velocity detection functioning as both the gaze direction detection means and the angular velocity detection means And part 7. The imaging unit 30, the see-through image display unit 6, and the line-of-sight direction / angular velocity detection unit 7 are all connected to the control / recording unit 4 via the cable 3.

  The see-through image display unit 6 includes an LED driver 112, an LED 113, a condenser lens 114, an LCD 115, an LCD driver 117, an HOE 116, an HOE 25, an LED 16a for projection, an LED driver 118, and a condenser lens. 16b, the switch 39, and the fourth CPU 111. In this embodiment, the see-through image display unit 6 is disposed only on the side of one eye of the observer (here, the right eye as a specific example). Therefore, the observer observes the see-through image only on the one eye side.

  The LED driver 112 causes the LED 113 to emit light based on the control of the fourth CPU 111.

  The LED 113 is a light emitting source and constitutes a projecting unit, and is driven by the LED driver 112 to emit light. The LED 113 includes, for example, diodes capable of emitting light of three colors of R (red), G (green), and B (blue).

  The condensing lens 114 constitutes the projection means, and condenses the light emitted by the LED 113.

  The LCD 115 constitutes the projection means, and is for displaying the photographed image frame, the photographed image, etc., and is illuminated from the back side by the light of the LED 113 through the condenser lens 114. This is a transmissive liquid crystal display means.

  The LCD driver 117 drives the LCD 115 based on the control of the fourth CPU 111 to display a photographic image frame and the like, and also serves as a correcting means for correcting parallax as will be described later.

  The HOE 116 is a reflective optical member that reflects light emitted through the LCD 115 downwardly (see FIGS. 13 and 14) while correcting aberrations as will be described later.

  The HOE 25 functions as a reflection type combiner, and the light from the HOE 116 is reflected and diffracted toward the photographer's eyes so that the image frame displayed on the LCD 115 can be observed. The combiner is configured to project and transmit external light toward the photographer's eyes. The HOE 25 of the first embodiment is similarly configured to have a minimum size in accordance with the transparent optical member 15 configured to have a minimum size.

  The light projecting LED 16a is included in the light projecting light emitting unit 16 for performing the distance measurement, and is a light emission source that emits light for distance measurement.

  The condensing lens 16b is for projecting the distance measuring light emitted from the light projecting LED 16a toward the subject.

  The LED driver 118 is for driving the light projecting LED 16 a based on the control of the fourth CPU 111.

  The switch 39 is connected to the fourth CPU 111. When the fourth CPU 111 detects that the switch 39 is closed, the fourth CPU 111 prohibits the display of the see-through image display unit 6. . The camera 1 is configured to be able to shoot while performing normal actions without being restricted by the shooting operation, and thus tends to be used, for example, when walking or driving a car. It is. However, if the photographer is in such a state and the see-through display is performed, the display may be distracted. Therefore, in order to prevent this, a switch 39 is provided so that see-through display can be prohibited. At this time, display is prohibited, but shooting itself can be continued.

  The fourth CPU 111 is a control unit that controls each circuit in the see-through image display unit 6. The fourth CPU 111 is bi-directionally connected to a third CPU 103 (to be described later) of the imaging unit 30 to control the circuit. The operation is to be performed.

  The operation of the see-through image display unit 6 is almost as follows.

  The fourth CPU 111 causes the LED 113 to emit light via the LED driver 112.

  The light emitted from the LED 113 is condensed by the condenser lens 114 and illuminates the LCD 115 from the back.

  The LED 113 displays only the G (green) diode, for example, among the diodes that emit light of the three colors R (red), G (green), and B (blue) when displaying a photographic image frame. Make it emit light.

  When the fourth CPU 111 generates a signal corresponding to the shooting image frame indicating the shooting range and transfers it to the LCD driver 117, and also transfers a control signal for causing light emission to the LED driver 112, the LCD driver 117 performs the LCD 115 based on the signal. To display the photographic image frame on the display surface, and the LED 113 emits light to illuminate the LCD 115 from the back side.

  The image of the photographic image frame projected from the LCD 115 thus illuminated is reflected vertically downward by the HOE 116 while being corrected for aberrations, and projected onto the HOE 25.

  The HOE 25 reflects the light beam from the HOE 116 toward the photographer's eyes. Thereby, the photographer can observe the photographed image frame indicating the photographing range as a virtual image.

  On the other hand, in the case of performing distance measurement, the fourth CPU 111 transfers a control signal for performing light emission for distance measurement to the LED driver 118.

  Upon receiving the control signal, the LED driver 118 causes the light projecting LED 16a to emit light. The distance measuring light emitted from the light projecting LED 16a is converted into parallel light by the condenser lens 16b and projected toward the subject.

  The illumination light thus projected is reflected by the subject, the reflected light is received by the imaging unit 30, and a distance measurement calculation is performed as described later.

  Next, the line-of-sight direction / angular velocity detection unit 7 is disposed on one eye (here, the left eye as a specific example) opposite to the side where the see-through image display unit 6 is disposed. It has become. The line-of-sight direction / angular velocity detection unit 7 is roughly classified into a line-of-sight direction detection unit that functions as a line-of-sight direction detection device and an angular velocity detection unit that functions as an angular velocity detection device.

  The line-of-sight direction detection unit is a line-of-sight direction detection unit, and includes an LED driver 124, an LED 125, a condenser lens 126, a reflection mirror 127, a half mirror 128, an HOE 24, an HOE 129, a reflection mirror 131, The imaging lens 132, the band pass filter 133, the CCD 134, the CCD driver 135, the CDS / AGC circuit 136, the A / D conversion circuit 137, and the TG 138 are configured. Among these, the LED driver 124, the LED 125, the condensing lens 126, the reflection mirror 127, and the HOE 24 constitute a light projecting system as an infrared light projecting unit for projecting an infrared parallel light beam to the eyes of the observer. ing. The above-mentioned HOE 24, half mirror 128, HOE 129, reflection mirror 131, imaging lens 132, bandpass filter 133, and CCD 134 constitute a light receiving system for receiving the light beam reflected from the eyes of the observer. doing.

  The angular velocity detection unit is angular velocity detection means, and includes angular velocity sensors 141 and 142, amplifiers 143 and 144, and an A / D conversion circuit 145.

  The second CPU 121 is provided to control the entire line-of-sight direction / angular velocity detector 7 including the line-of-sight direction detector and the angular velocity detector.

  The LED driver 124 is for driving the LED 125 based on the control of the second CPU 121.

  The LED 125 is an infrared light-emitting diode that is driven by an LED driver 124 and emits infrared light for detecting a visual line direction.

  The condensing lens 126 converts infrared light emitted from the LED 125 into a parallel light beam.

  The reflection mirror 127 is a reflection optical member that reflects the infrared light converted into a parallel light beam by the condenser lens 126 downward (see FIGS. 15 and 16).

  The half mirror 128 transmits infrared light from the reflection mirror 127 toward the HOE 24 and reflects light from the HOE 24 in the horizontal direction.

  The HOE 24 reflects infrared light transmitted through the half mirror 128 toward the observer's eye and reflects light from the observer's eye toward the half mirror 128. The HOE 24 of the first embodiment is similarly configured to have a minimum size in accordance with the transparent optical member 14 configured to have a minimum size. This HOE 24 has a wavelength selectivity for a narrow predetermined band in the infrared region, and exhibits high reflection characteristics only for infrared light in the selected band, while against light beams of other wavelengths. Shows high transmission characteristics. As described above, the HOE 24 has a wavelength selection function in a completely different band from that of the HOE 25 of the see-through image display unit 6, but the overall luminous transmission characteristics (or average luminous transmission characteristics) are as follows. It is almost the same as the HOE 25. In this way, by arranging optical elements that have the same luminous transmission characteristics on the left and right eyes on average, there is no sense of incongruity, and it reduces eye fatigue even when used for a long time. Can do.

  The HOE 129 reflects the light beam from the observer's eye, which is guided by the HOE 24 and the half mirror 128, vertically upward. The HOE 129 has the same wavelength selectivity as the HOE 24. Therefore, by passing through the HOE 129, light beams having wavelengths other than the infrared light included in the predetermined band in the light flux incident from the observer's eye side are cut (that is, transmitted without being reflected). Can be). Thus, it is possible to detect a Purkinje image as described later with higher accuracy while reducing the influence of noise caused by light beams in other wavelength regions. Note that the use of HOE 129 as an optical element not only has the advantage of increasing wavelength selectivity, but also has transparency to light other than infrared light, so that it is disposed in the transparent optical member 14. There is also an advantage that it is inconspicuous.

  The reflection mirror 131 reflects the light beam from the HOE 129 in the horizontal direction.

  The imaging lens 132 is for imaging the light beam reflected by the reflection mirror 131 on the imaging surface of the CCD 134.

  The bandpass filter 133 transmits only the infrared light in the predetermined band out of the light beam formed by the imaging lens 132. As described above, the band is already limited by the HOE 129. However, when the HOE 129 is disposed in the transparent optical member 14, other visible light may be mixed. This bandpass filter 133 limits the band again. Thus, by providing the band pass filter 133 on the front side of the CCD 134, it is possible to further reduce the influence of external light noise.

  The CCD 134 is a two-dimensional photoelectric conversion means, and is configured as an image pickup element having a two-dimensional image pickup surface. The CCD 134 photoelectrically converts the image of the observer's eye formed as described above, It is output as an electrical signal. Based on the output of the CCD 134, the position of the Purkinje image and the position of the pupil center are obtained as will be described later, and the line-of-sight direction is calculated based on their relative positional relationship.

  The CCD driver 135 is for driving the CCD 134 based on the control of the second CPU 121.

  The CDS / AGC circuit 136 performs noise removal and amplification processing on the video signal output from the CCD 134.

  The TG 138 supplies timing signals for operating the CCD driver 135 and the CDS / AGC circuit 136 in association with each other based on the control of the second CPU 121.

  The A / D conversion circuit 137 converts the analog video signal output from the CDS / AGC circuit 136 into a digital video signal. The digital video signal converted by the A / D conversion circuit 137 is output to a multiplexing circuit 104 described later of the imaging unit 30.

  The angular velocity sensors 141 and 142 are stored in the storage unit 27 as described above, and are for detecting angular velocities in mutually independent directions (for example, the yaw direction and the pitch direction).

  The amplifiers 143 and 144 are for amplifying the outputs of the angular velocity sensors 141 and 142, respectively.

  The A / D conversion circuit 145 converts analog signals from the angular velocity sensors 141 and 142 amplified by the amplifiers 143 and 144, respectively, into digital data. The digital data converted by the A / D conversion circuit 145 is output to the second CPU 121.

  The second CPU 121 is a control means for controlling each circuit in the line-of-sight direction / angular velocity detection unit 7 and is connected bidirectionally to a third CPU 103 (to be described later) of the imaging unit 30 while cooperating with it. The control operation is performed. The second CPU 121 includes a RAM 122 functioning as a memory and a timer 123 for measuring time.

  The operation principle, action, and the like of the line-of-sight direction / angular velocity detection unit 7 will be described later with reference to other drawings.

  The imaging unit 30 includes a photographing optical system 31, a low-pass filter 86, a CCD 87, a CDS / AGC circuit 88, an A / D conversion circuit 89, a TG (timing generator) 90, a CCD driver 91, and a USM driver. 95, aperture shutter driver 96, AE processing circuit 97, AF processing circuit 98, the first microphone 17, the second microphone 18, the amplification circuit 99, the amplification circuit 100, and the A / D conversion circuit. 101, EEPROM 102, multiplexing circuit 104, and third CPU 103.

  The photographing optical system 31 is for forming an optical subject image, and is configured as a zoom optical system having a variable focal length.

  The low-pass filter 86 is for removing unnecessary high-frequency components from the light beam that has passed through the photographing optical system 31.

  The CCD 87 is an image pickup device, and converts the optical subject image formed by the photographing optical system 31 through the low-pass filter 86 into an electrical signal and outputs it.

  The CDS / AGC circuit 88 is a signal processing means for performing noise removal and amplification processing as will be described later on the signal output from the CCD 87.

  The A / D conversion circuit 89 is signal processing means for converting an analog image signal output from the CDS / AGC circuit 88 into a digital image signal.

  The CCD driver 91 is for controlling and driving the CCD 87.

  The TG (timing generator) 90 supplies signals for controlling the timing to the CDS / AGC circuit 88, the A / D conversion circuit 89, and the CCD driver 91, respectively. The TG 90 is connected to and controlled by the second CPU 121 of the line-of-sight direction / angular velocity detector 7 in a bidirectional manner.

  The USM driver 95 is a drive circuit for selectively driving USMs (Ultra Sonic Motors) 92, 93, 94, which will be described later, included in the photographing optical system 31. The USM driver 95 is also controlled by the second CPU 121 of the line-of-sight direction / angular velocity detection unit 7.

  The diaphragm shutter driver 96 is a drive circuit for controlling and driving a diaphragm shutter 84 (described later) included in the photographing optical system 31. The aperture shutter driver 96 is controlled by the second CPU 121 of the line-of-sight direction / angular velocity detector 7.

  The AE processing circuit 97 is an auto exposure processing circuit that performs calculation for exposure control based on the output of the A / D conversion circuit 89, and outputs the calculation result to the third CPU 103.

  The AF processing circuit 98 is an autofocus processing circuit that performs calculation for autofocus (AF) control based on the output of the A / D conversion circuit 89, and outputs the calculation result to the third CPU 103. .

  The first microphone 17 and the second microphone 18 are for recording audio from the subject side in stereo as described above.

  The amplifier circuit 99 and the amplifier circuit 100 are for amplifying audio signals input from the first microphone 17 and the second microphone 18, respectively.

  The A / D conversion circuit 101 converts the analog audio signal amplified by the amplification circuit 99 and the amplification circuit 100 into a digital audio signal and outputs the digital audio signal to the third CPU 103.

  The EEPROM 102 records various correction data for exposure control, autofocus processing, and the like when the camera is manufactured. The data recorded in the EEPROM 102 is read out by the third CPU 103.

  The multiplexing circuit 104 includes a video signal output from the A / D conversion circuit 89 and a video signal relating to a photographer's eye output from the A / D conversion circuit 137 of the visual angle direction / angular velocity detection unit 7 ( One of the biological information), angular velocity information (one of the biological information) related to the photographer's head output from the second CPU 121 of the viewing angle direction / angular velocity detection unit 7, and the third CPU 103. Based on the timer information output from the second CPU 121, the audio data is multiplexed so that the video signal generated at the same time and the biological information are related to each other, and output to the control / recording unit 4. To do.

  The third CPU 103 is a control means for controlling each circuit in the imaging unit 30, and cooperates with the fourth CPU 111 of the see-through image display unit 6 and the second CPU 121 of the visual angle direction / angular velocity detection unit 7. A control operation is performed. Further, the third CPU 103 is controlled by bidirectionally communicating with a first CPU 161 (to be described later) of the control / recording unit 4.

  More specifically, the photographing optical system 31 includes a front lens 81, a variator lens 82, a compensator lens 83, an aperture shutter 84, a focus lens 85, and USMs 92, 93, and 94. Yes.

  The front lens 81 is located closest to the subject among the plurality of lenses included in the photographing optical system 31.

  The variator lens 82 is for changing the focal length of the photographing optical system 31.

  The compensator lens 83 is used to correct a focus position shift caused by changing the focal length of the photographing optical system 31 by the variator lens 82.

  The diaphragm shutter 84 serves both as a diaphragm function for defining the passage range of the light beam passing through the photographing optical system 31 and a shutter function for defining the passage time of the light beam.

  The focus lens 85 is for adjusting the focus of the photographic optical system 31. When the focus is adjusted, the subject image is focused on the CCD 87 and formed.

  The USMs 92, 93, and 94 are driving sources for driving the variator lens 82, the compensator lens 83, and the focus lens 85, respectively.

  The control / recording unit 4 includes a separation circuit 151, a DSP circuit 152, a delay circuit 153, a multiplexing circuit 154, a memory 155, a compression / expansion circuit 156, a recording memory 157, and a D / A conversion. The circuit 158, the LCD 48 (see FIG. 10), the LCD driver 160, the first operation switch 162, the receiving circuit 163, the power supply circuit 164, and the first CPU 161 are configured.

  The separation circuit 151 separates the signal transmitted by the multiplexing circuit 104 into image data and other data again. The separation circuit 151 outputs image data to the DSP circuit 152 and other data to the delay circuit 153.

  The DSP circuit 152 performs predetermined digital signal processing on the image data from the separation circuit 151.

  The delay circuit 153 is for delaying data input from the separation circuit 151 in order to achieve temporal synchronization with the image data processed by the DSP circuit 152.

  The multiplexing circuit 154 multiplexes the image data processed by the DSP circuit 152 and the data transmitted from the delay circuit 153.

  The memory 155 temporarily stores the signal from the multiplexing circuit 154, and is composed of, for example, a frame buffer.

The compression / decompression circuit 156 compresses the digital signal stored in the memory 155 and also decompresses the compressed digital signal read from the recording memory 157. The recording memory 157 includes: It serves as both image data recording means and biometric information recording means, and records the digital signal compressed by the compression / expansion circuit 156.

  The D / A conversion circuit 158 converts the digital signal stored in the memory 155 into an analog signal.

  The LCD 48 displays an image based on the analog image signal converted by the D / A conversion circuit 158.

  The LCD driver 160 is for controlling and driving the LCD 48.

  The first operation switch 162 is used to perform various operation inputs related to the camera 1, and includes each switch as shown in FIG.

  The receiving circuit 163 is for receiving a signal from a transmitting circuit 173 described later of the remote controller 5.

  The power supply circuit 164 includes, for example, a battery configured to be detachable, and is not only configured to supply power to the control / recording unit 4 but also to the head mounting unit 2. It is configured to be able to supply power.

  The first CPU 161 is an integrated control unit related to the camera 1, and controls each circuit in the control / recording unit 4 and communicates with the third CPU 103 to communicate with the third CPU 103. It is also configured to perform control.

  The remote control unit 5 includes a second operation switch 171, a decoder 172, a transmission circuit 173, and a power supply circuit 174.

  The second operation switch 171 includes switches as shown in FIG.

  The decoder 172 is for encoding the operation input from the second operation switch 171 into a signal for wireless transmission.

  The transmission circuit 173 is for transmitting the signal encoded by the decoder 172 to the reception circuit 163 of the control / recording unit 4. The first CPU 161 of the control / recording unit 4 performs predetermined processing on the information of the operation switch received by the receiving circuit 163 and performs predetermined control on the head-mounted unit 2 via the third CPU 103. .

  The power supply circuit 174 is for supplying power to each circuit in the remote control unit 5 and includes a battery and the like.

  The operation of the camera 1 is almost as follows.

  The light beam that has passed through the photographing optical system 31 forms an image on the imaging surface of the CCD 87 via the low-pass filter 86.

  When a moving image recording operation is performed by the first operation switch 162 of the control / recording unit 4 or a still image shooting operation is performed by the release switch 73 of the remote control unit 5, the subject image is photoelectrically converted by the CCD 87. The analog image signal is output after conversion.

  The image signal from the CCD 87 is input to the CDS / AGC circuit 88, and the CDS circuit unit in the CDS / AGC circuit 88 performs known correlated double sampling to remove reset noise, and The signal is amplified to a predetermined signal level by the AGC circuit unit in the CDS / AGC circuit 88 and output.

  The analog image signal from the CDS / AGC circuit 88 is converted into a digital image signal (image data) by the subsequent A / D conversion circuit 89. In this embodiment, the output signal of the A / D conversion circuit 89 is referred to as RAW image data. That is, the RAW image data in this embodiment is defined as digital data immediately after the analog output signal from the CCD 87 is first A / D converted, and is data before other digital signal processing or the like is performed.

  The timing control signal generated by the TG 90 is inputted to the CDS / AGC circuit 88 and the A / D conversion circuit 89. The timing control signal from the TG 90 is further inputted to the CCD driver 91. Is also entered.

  On the other hand, the output signals from the first microphone 17 and the second microphone 18 are amplified by the amplification circuits 99 and 100, respectively, and then converted into digital data by the A / D conversion circuit 101 in a time-division manner at a predetermined sampling period. And transferred to the third CPU 103. The third CPU 103 transfers the audio data converted into the digital data to the multiplexing circuit 104 at a predetermined timing.

  Image data which is an output signal from the A / D conversion circuit 89, audio data from the first microphone 17 and the second microphone 18, and gaze direction data (gaze direction information) from the gaze direction / angular velocity detection unit 7. ) And the angular velocity data of the photographer's head and the timer information counted by the timer 123 of the second CPU are multiplexed by the multiplexing circuit 104.

  As shown in FIG. 42, the signal multiplexed by the multiplexing circuit 104 is recorded with the detection start time at which each data is detected at the head, and then various data such as the image data and audio data are recorded. The signals are alternately output at predetermined intervals. FIG. 42 is a diagram showing a time-series configuration of a signal output from the multiplexing circuit 104. In FIG. For example, assuming that the capture cycle of image data, audio data, and angular velocity data is 1/30 seconds each, the capture cycle of gaze direction data is 1 second, and the data for 1 second is one unit, As shown in FIG. 42, 30 sets of data including three sets of image data, audio data, and angular velocity data are repeatedly recorded starting from the start time data, and finally, the line-of-sight direction data is recorded. The Note that the order of the data shown in FIG. 42 is merely an example, and therefore, for example, the line-of-sight direction data may be recorded immediately after the time data.

  A plurality of unit-unit data including such digitized data is output to the control / recording unit 4 via the cable 3.

  As described above, the imaging unit 30 performs analog signal processing of the image signal generated by the CCD 87 and converts the image data into a digital signal and outputs the digital signal. Therefore, the analog signal is output from the imaging unit 30. Is not output to the outside. Therefore, the configuration is strong against external noise that may be received when an image signal is transmitted through the cable 3 or the like.

  Further, since the imaging unit 30 outputs RAW image data, it is not necessary to provide a signal processing circuit such as color separation or white balance adjustment inside the imaging unit 30, and the imaging unit 30 It is possible to reduce the size and weight of the provided head mounting portion 2.

  The signal transmitted from the multiplexing circuit 104 to the control / recording unit 4 is again separated into image data and other data by the separation circuit 151 in the control / recording unit 4.

  The image data separated by the separation circuit 151 is subjected to a predetermined image processing calculation in the DSP circuit 152, and an auto white balance process is also performed on the image data based on the obtained calculation result.

  On the other hand, data other than the image data separated by the separation circuit 151 (that is, data including line-of-sight direction data, angular velocity data, audio data, etc.) is temporally related to the image data processed by the DSP circuit 152 by the delay circuit 153. Delayed to ensure proper synchronization.

  The output from the DSP circuit 152 and the output from the delay circuit 153 are input to the multiplexing circuit 154, multiplexed by the multiplexing circuit 154, and the multiplexed signal is temporarily stored in the memory 155. To be recorded.

  Of the various data recorded in the memory 155, the image data and audio data are each compressed in a predetermined format by the compression circuit unit included in the compression / decompression circuit 156, and then recorded in the recording memory 157. Is done.

  On the other hand, time data, angular velocity data, and line-of-sight direction data out of various data recorded in the memory 155 are directly recorded in the recording memory 157. Of these, the angular velocity data and the line-of-sight direction data may be recorded in the recording memory 157 after being subjected to a known reversible compression process.

  As the recording memory 157, various recording media such as a card memory and a disk memory such as a DVD can be widely applied. The outline of the recording format related to various data as described above will be described later.

  Further, a recorded image is selected by operating the menu button 63 of the first operation switch 162, the menu selection switches 66, 67, 68, 69, the confirmation switch 65, and the like, and is played back by operating the playback / stop switch 59. Is compressed, the compressed data stored in the recording memory 157 is expanded by the expansion circuit unit in the compression / expansion circuit 156 and temporarily stored in the memory 155, and the image The data is converted into an analog image signal by the D / A conversion circuit 158 and then displayed on the LCD 48. The operation of the LCD 48 at this time is controlled by a signal generated from the LCD driver 160.

  On the other hand, the digital image data from the A / D conversion circuit 89 is input to the AE processing circuit 97 and the AF processing circuit 98 in the imaging unit 30.

  The AE processing circuit 97 calculates and calculates the AE evaluation value corresponding to the brightness of the subject by performing processing such as calculating the luminance value of image data for one frame (one screen) and performing weighted addition. The result is output to the second CPU 121 via the third CPU 103. The second CPU 121 controls the exposure time of the CCD 87 and the diaphragm shutter 84 based on the AE evaluation value.

  Further, the AF processing circuit 98 extracts a high frequency component from the luminance component of the image data for one frame (one screen) using a high-pass filter or the like based on the control of the second CPU 121 via the third CPU 103. An AF evaluation value corresponding to a high frequency region side contour component or the like is calculated by calculating a cumulative addition value of the high frequency component, and the calculation result is output to the second CPU 121 via the third CPU 103. The second CPU 121 drives the focus lens 85 via the USM driver 95 based on the AF evaluation value calculated by the AF processing circuit 98, and performs focus detection to obtain a focused state. .

  The EEPROM 102 records various correction data necessary for exposure control, autofocus processing, and the like when the camera is manufactured. The second CPU 121 reads correction data from the EEPROM 102 via the third CPU 103 as necessary. Various operations are performed.

  Next, a mainly optical configuration of the see-through image display unit 6 will be described with reference to FIGS. 12 is a diagram for explaining the principle of the optical system of the see-through image display unit, FIG. 13 is a front view including a partial cross section showing the configuration of the optical system of the see-through image display unit, and FIG. 14 is an optical diagram of the see-through image display unit. It is a left view which shows the structure of a type | system | group.

  This see-through image display unit 6 can superimpose and display a photographic image frame indicating a photographic range as a virtual image on a subject that the photographer is actually directly observing. Hereinafter, the display is referred to as see-through display. Note that “substantially directly observing” means not only when observing with the naked eye, but also when observing through a substantially flat transparent member formed of glass or plastic, or This includes the case of observation through a diopter adjustment lens.

  First, the principle of displaying a see-through image by the optical system of the see-through image display unit 6 in the first embodiment (hereinafter referred to as “see-through image display optical system”) will be described with reference to FIG.

  The light emitted from the LED 113 is condensed by the condenser lens 114 and illuminates the LCD 115 from the back. As described above, the LED 113 emits only a G (green) diode, for example, when displaying a photographic image frame.

  The fourth CPU 111 generates a signal corresponding to the shooting image frame indicating the shooting range and outputs the signal to the LCD driver 117. The LCD driver 117 drives the LCD 115 based on this signal to display a photographic image frame on the LCD 115.

  The image of the photographic image frame received from the LED 113 and emitted from the LCD 115 is reflected by the HOE 25 and then guided to the photographer's eyes. Thus, the photographer can observe the photographic image frame indicating the photographic range as the virtual image VI. In FIG. 12, since the principle is described, the illustration of the HOE 116 is omitted.

  The HOE 25 is a volume phase type holographic optical element using a photosensitive material such as a photopolymer or dichromated gelatin, and reflects light with the maximum reflectance at each wavelength of R, G, B emitted from the LED 113. Designed to have the characteristics to Therefore, when G light is emitted when displaying the photographic image frame, the green photographic image frame is clearly displayed as a virtual image. HOE has excellent wavelength selectivity, and exhibits high reflection characteristics in the extremely narrow wavelength range for the light beams of the above-described R, G, and B wavelengths, while it has a high reflection characteristic for light beams of other wavelengths. Show high transmission characteristics. Accordingly, external light in the same wavelength region as the display light is diffracted and reflected and does not reach the photographer's pupil, but external light in other wavelength regions reaches the photographer's pupil. In general, since visible light has a wide wavelength bandwidth, it is possible to observe an external field image without any trouble even if light having an extremely narrow wavelength width including R, G, and B wavelengths does not arrive. is there.

  In addition, the see-through image display unit 6 can display the image picked up by the image pickup unit 30 as a color image in a see-through manner. In this case, the LCD 115 displays a picked-up image, The LED 113 may emit light of R, G, and B colors. As a result, the captured image reaches the photographer's pupil as a virtual image from the HOE 25.

  The HOE 116 has a function of not only reflecting the light from the LCD 115 so as to guide it to the HOE 25 but also correcting the image plane distortion. Although the HOE 116 is used here, a free-form optical element can be used instead. Since a free-form optical element is small and light, it can correct complex aberrations, and thus can display a clear image with little aberration without increasing the weight.

  Subsequently, a specific arrangement example of the see-through image display optical system will be described with reference to FIGS. 13 and 14.

  The LED 113, the condensing lens 114, the LCD 115, and the HOE 116 are arranged in this order at the position above the transparent optical member 15 inside the frame portion 13 as shown in FIG. Each of these members is fixed so as to be sandwiched between holding frames provided in the frame portion 13. At this time, the LED 113 is fixed by the holding frame while being mounted on the electric circuit board 181. Further, of these, the HOE 116 is disposed so as to be inclined so as to reflect the light from the LED 113 vertically downward as described above.

  As shown in FIG. 14, the transparent optical member 15 includes light guide members 182 and 183 formed with transparent glass, plastic, or the like so as to have a predetermined thickness, and between these light guide members 182 and 183. The HOE 25 is disposed so as to be inclined so as to reflect light backward while being sandwiched. In such a configuration, the light reflected from the HOE 116 passes through the inside of the light guide member 182 disposed on the upper side of the HOE 25 and reaches the HOE 25. The light propagation inside the light guide member 182 may be only transmission as shown in FIG. 14, or may be a combination of transmission and total reflection on the inner surface. When an optical design that combines transmission and total reflection is performed, the transparent optical member 15 can be made thin, so that the weight of the head mounting portion 2 can be further reduced.

  The see-through image display optical system includes the LED 113, the condenser lens 114, the LCD 115, the HOE 116, the HOE 25, and the light guide members 182 and 183 among the above-described members. ing.

  Next, an example of the configuration of the optical system of the line-of-sight direction detection unit in the line-of-sight direction / angular velocity detection unit 7 will be described with reference to FIGS. 15 and 16. FIG. 15 is a front view including a partial cross section showing a configuration example of the optical system of the line-of-sight direction detection unit, and FIG. 16 is a left side view showing a configuration example of the optical system of the line-of-sight direction detection unit.

The LED 125, the condenser lens 126, the reflection mirror 127, the reflection mirror 131, the imaging lens 132, the band pass filter 133, and the CCD 134 are arranged at a position above the transparent optical member 14 in the frame portion 13. As shown in FIG.

  A half mirror 128 and a HOE 24 are disposed in the transparent optical member 14 below the reflection mirror 127. Further, an HOE 129 is disposed in the transparent optical member 14 on the side of the half mirror 128.

  In such a configuration, when infrared light is emitted from the LED 125, it is converted into a parallel light beam by the condenser lens 126 and then reflected vertically downward by the reflection mirror 127.

  The infrared light reflected by the reflecting mirror 127 enters the transparent optical member 14, passes through the half mirror 128 disposed in the transparent optical member 14, and is directed toward the observer's eyes by the HOE 24. Reflected.

  On the other hand, the infrared light reflected from the observer's eyes is reflected upward by the HOE 24 and further reflected laterally by the half mirror 128. The reflected light is further reflected upward by the HOE 129 and reaches the reflecting mirror 131 disposed in the frame portion 13.

  The light reflected to the side by the reflecting mirror 131 passes through the imaging lens 132 and the band-pass filter 133, and the image of the eye of the observer relating to the wavelength range of the infrared light reflected by the HOE 24. As shown in FIG.

  The image signal converted by the CCD 134 is transferred to the control / recording unit 4 via the CDS / AGC circuit 136, the A / D conversion circuit 137, the multiplexing circuit 104, etc. In the first CPU 161, as will be described later, the position of the Purkinje image and the center position of the pupil are obtained, and the line-of-sight direction is obtained from the relative relationship therebetween.

  The half mirror 128 and the HOE 129 are arranged in the transparent optical member 14 as described above, and the position at this time is the field of view of an observer who observes the subject wearing the head mounting portion 2. It is desirable to have an upper portion that is difficult to enter.

  According to the configuration shown in FIGS. 15 and 16, the HOE 24 having high wavelength selectivity and high reflectance with respect to light in the selected wavelength band is used, and bands having similar wavelength selectivity are used. Since the pass filter 133 is disposed in front of the CCD 134, a signal with a high S / N ratio can be obtained.

  With such a configuration, it is possible to accurately determine the line-of-sight direction of the observer who is observing the subject.

  Next, another example of the configuration of the optical system of the visual line direction detection unit in the visual line direction / angular velocity detection unit 7 will be described with reference to FIGS. 17 and 18. FIG. 17 is a front view including a partial cross section showing another configuration example of the optical system of the line-of-sight direction detection unit, and FIG. 18 is a left side view showing another configuration example of the optical system of the line-of-sight direction detection unit.

  In the configuration shown in FIGS. 15 and 16, since the half mirror 128 and the HOE 129 are directly exposed to outside light, it is difficult to completely block outside light noise from reaching the CCD 134. is there. On the other hand, in the configuration as shown in FIGS. 17 and 18, an optical path for projecting infrared light to the observer's eye and an optical path for receiving infrared light reflected by the observer's eye by the CCD 134. Are completely separated from each other.

  That is, in the optical system of the line-of-sight direction detection unit, the half mirror 128 and the HOE 129 are omitted, and the HOE 24 is configured to be slightly longer in the left-right direction than that shown in FIG.

  The infrared light reflected by the reflection mirror 127 is incident obliquely toward the left side portion of the HOE 24, for example. The HOE 24 projects the infrared light from the reflection mirror 127 toward the observer's eyes so as to be slightly inclined in the horizontal direction.

  On the other hand, the reflected light from the observer's eyes is received by, for example, the right side portion of the HOE 24 and reflected almost vertically upward toward the reflection mirror 131. The configuration after the reflecting mirror 131 and the like are the same as those shown in FIGS.

  According to the configuration shown in FIGS. 17 and 18, the HOE 24 having high wavelength selectivity and high reflectance for light in the selected wavelength band is used, and bands having similar wavelength selectivity are used. Since the pass filter 133 is disposed in front of the CCD 134, a signal with a high S / N ratio can be obtained. Further, since only the HOE 24 is arranged on the transparent optical member 14 to which the external light is irradiated, compared to the configuration of FIGS. 15 and 16 in which the half mirror 128 and the HOE 129 are arranged on the transparent optical member 14. Thus, the influence of external light noise can be further reduced.

  The propagation of light inside the transparent optical member 14 may be only transmission as shown in FIGS. 16 and 18, but the transmission and transmission on the inner surface are similar to the propagation of light inside the transparent optical member 15. It may be a combination of total reflection. In the case of performing an optical design that combines this transmission and total reflection, the transparent optical member 14 can be made thin, and the weight of the head mounting portion 2 can be further reduced.

  Next, a configuration in which the imaging unit 30 is attached to the side surface of the frame unit 13 so that the angle can be relatively adjusted in the pitch direction and the yaw direction will be described with reference to FIGS. 19 and 20. FIG. 19 is a plan view and a right side view showing a configuration for attaching the imaging unit 30 to the frame unit 13, and FIG. 20 is a right side view showing a configuration of holes provided in the frame unit 13 for attaching the imaging unit 30.

  The head-mounted camera 1 of the present embodiment is one in which a photographer designates a photographing image frame indicating a photographing range and performs photographing at an angle of view corresponding to the viewing angle of the designated photographing image frame. It is necessary to correct the lux. The cause of this parallax is the horizontal positional deviation between the photographer's visual axis and the photographic optical axis, and the angular deviation between the visual axis and the photographic optical axis. Therefore, an adjustment mechanism (adjustment means) is provided that can accurately correct this angular deviation.

  As shown in FIG. 19A, the frame portion 13 and the temple portion 12 are connected to each other via a hinge 78 so as to be foldable. The hinge 78 is disposed at a position slightly further away from the front portion 11 side than the hinge 79 on the right eye side via a slightly longer joint 29 extending from the frame portion 13.

  An adjustment mechanism having a shape portion 33a along the side surface and a shape portion 33b erected substantially perpendicularly from the side surface on the side surface of the joint 29 and having a substantially L shape when viewed from the front. (Adjustment means) A pedestal 33 is connected. The adjustment mechanism is a mechanism for adjusting the relative angle between the front unit 11 and the imaging unit 30 in a head-mounted camera adjustment apparatus to which the head-mounted camera adjustment method is applied. By using the adjustment mechanism, it is possible to adjust the optical axis and the visual axis of the photographing optical system 31 included in the imaging unit 30.

  That is, as shown in FIG. 20, the joint 29 has a hole 191 as a pitch direction adjusting means on the front side, and a long hole 192 as a pitch direction adjusting means having an arc shape around the hole 191 on the rear side. , Respectively. The pedestal 33 is attached to the joint 29 by screwing the screws 34 and 35 serving as pitch direction adjusting means into the shape portion 33 a of the pedestal 33 through these holes 191 and 192.

  Further, in the shape portion 33b of the pedestal 33, as shown in FIG. 19A, a hole 193 serving as a yaw direction adjusting means is formed on the front side, and a yaw direction having an arc shape centered on the hole 193 is formed on the rear side. Long holes 194 serving as adjusting means are respectively formed. Through these holes 193 and 194, as shown in FIG. 19B, screws 36 and 37, which are yaw direction adjusting means, are screwed to the bottom surface side of the imaging unit 30, respectively, so that the imaging unit 30 is mounted on the base. 33 is attached. A cable 38 extends from the back side of the imaging unit 30 and is bent to the subject side before being connected to an electrical circuit or the like in the frame unit 13.

  In such a configuration, by changing the position in the long hole 192 through which the screw 35 is inserted in a state where the screw 34 and the screw 35 are slightly loosened, the pedestal 33 rotates around the screw 34, and the pedestal 33 As a result, the angle of the imaging unit 30 attached to the pedestal 33 can be adjusted in the pitch direction. Thus, after the adjustment to the desired position, the screw 34 and the screw 35 may be tightened.

  Similarly, by changing the position in the long hole 194 through which the screw 37 is inserted while the screw 36 and the screw 37 are slightly loosened, the imaging unit 30 rotates with respect to the base 33 around the screw 36. The angle of the imaging unit 30 in the yaw direction can be adjusted. Thus, after the adjustment to the desired position, the screws 36 and 37 are similarly fastened.

  According to such a configuration, it is possible to adjust the relative angle between the see-through image display unit 6 and the imaging unit 30 in the pitch direction and the yaw direction. Further, since the imaging unit 30 is fixed to the front unit 11 via the pedestal, the imaging unit 30 is not folded even if the temple unit 12 is folded, and the imaging unit 30 and the see-through image display unit 6 are relatively relative to each other. The possibility that a large angular deviation will occur is reduced. Further, since the adjustment mechanism is simple, it can be configured at low cost.

  In the above description, the relative angle in the yaw direction between the imaging unit 30 and the pedestal 33 is adjusted, and the relative pitch direction angle between the joint 29 on the side surface of the frame 13 and the pedestal 33 is adjusted. However, on the contrary, by changing the mounting position of the imaging unit 30 with respect to the pedestal 33, the angle of the relative pitch direction between the imaging unit 30 and the pedestal 33 is adjusted, and the frame unit 13 with respect to the pedestal 33 is adjusted. By changing the attachment position, it is also possible to adjust the relative angle in the yaw direction between the joint 29 on the side surface of the frame portion 13 and the pedestal 33.

  Next, an example of displaying an image by the see-through image display unit 6 will be described with reference to FIGS.

  First, FIG. 21 is a diagram illustrating a display example of an initial state in the see-through display. When the camera 1 is turned on or the system is reset, the display as shown in FIG. 21 is performed. At this time, as shown in the drawing, a photographic image frame 201 indicating a photographing range corresponding to a standard lens (for example, an angle of view is 50 degrees) is displayed in a see-through manner (that is, as viewed from the photographer). A photographed image frame 201 with a viewing angle of 50 degrees is displayed in a see-through manner).

  Next, FIG. 22 is a diagram illustrating a display example when zooming to tele is performed. The displayed photographic image frame 201 indicates a photographic range corresponding to telephoto than shown in FIG. As described above, the change of the photographic image frame 201 is performed by, for example, operating the zoom switch 75. At this time, the photographic image angle of the imaging unit 30 is also matched with the viewing angle of the photographic image frame 201. The focal length of the optical system 31 is changed. Specifically, by operating the tele switch 75a of the zoom switch 75 in the photographing range corresponding to the focal length of the standard lens as shown in FIG. 21, the telephoto side as shown in FIG. Changes are made.

  Next, FIG. 23 is a diagram illustrating a display example when zoom to wide and exposure correction are performed. The displayed photographic image frame 201 indicates a photographic range corresponding to a wider range than that shown in FIG. 21, and the exposure correction amount is displayed as an information display 202 at the lower right of the photographic image frame 201, for example. It is displayed. In the example shown in this figure, it is shown that +1.0 exposure correction is performed by the exposure correction switch 76, for example. It should be noted that exposure correction is not limited to being indicated by numbers, and it is needless to say that various displays such as bar graphs and indices may be used. 23 is operated by operating the wide switch 75b of the zoom switch 75 in the shooting range corresponding to the focal length of the standard lens as shown in FIG. 21, for example. Is set.

  FIG. 24 is a diagram illustrating a display example when electronic view display is performed. For example, when the view mode (V) is selected by the F / V switch 72, as shown in this figure, the electronic image 203 picked up by the image pickup unit 30 is projected as a virtual image to the photographer's eyes. ing. Note that the size of the image displayed as the electronic view can be set according to the resolution of the image. For example, when the resolution is low, the image may be displayed small.

  FIG. 25 is a diagram illustrating a display example during video recording. For example, when the recording switch 74 is operated and recording is in progress, as shown in FIG. 25, a shooting image frame 201 indicating the shooting range is displayed and an information display 204 indicating that recording is in progress is displayed. The characters “REC” are displayed, for example, in the lower right of the photographic image frame 201. The display indicating that recording is in progress is not limited to characters, as described above.

  FIG. 26 is a diagram illustrating a display example in the manual mode. For example, when the manual mode (M) is set by operating the FA / A / M switch 71, the information display 205 indicating the manual mode (M) is displayed as “MANU” as a photographic image frame 201. For example, it is displayed in the lower right. On the other hand, when the “MANU” information display 205 is not performed, the auto mode (A) is set.

  FIG. 27 is a diagram illustrating a display example during video recording in the full auto mode. For example, when the full auto mode (FA) is set by operating the FA / A / M switch 71 and the recording switch 74 is operated and recording is in progress, as shown in FIG. A shooting frame 201 indicating the shooting range is displayed, an information display 206 indicating that the camera is in the full auto mode is displayed as “FA”, and an information display 207 indicating that recording is being performed is displayed as “REC”. For example, the characters are displayed at the lower right of the photographic image frame 201 as characters. The display indicating the full mode or recording is not limited to characters, as described above.

  FIG. 28 shows an example of a warning display 208 displayed in a see-through manner when the focal length f of the photographic optical system 31 has reached the lower limit value k1 at which the focus can be adjusted, and when it is about to be operated to a smaller value. FIG. That is, when the zoom operation to the wide side is performed and the wide end of the zoom is reached, when the zoom operation to the wide side is still being performed, the warning display 208 shows the shooting image frame 201 indicating the shooting range. It is done with the display.

  FIG. 29 shows an example of a warning display 209 that is displayed see-through when the focal length f of the photographic optical system 31 has reached the upper limit value k2 at which the focus can be adjusted, and is about to be operated further. FIG. That is, when the zoom operation to the tele side is performed and the tele end of the zoom is reached, when the zoom operation to the tele side is still being performed, this warning display 209 indicates the shooting image frame 201 indicating the shooting range. It is done with the display.

  FIG. 30 is a diagram illustrating a display example when an operation for capturing a still image is performed. At this time, the photographic image frame 201 indicating the photographic range is displayed, and the information display 210 indicating that the still image has been recorded is displayed as, for example, the lower right of the photographic image frame 201 as the characters “REL”. It has become. The display indicating that this still image has been recorded is not limited to characters, as described above.

  FIG. 31 is a diagram illustrating a display example in the calibration mode. For example, when the calibration mode is selected by operating the menu button 63 of the first operation switch 162, the menu selection switches 66, 67, 68, 69, the confirmation switch 65, etc., the display as shown in FIG. Is supposed to do. That is, here, the information display 211 indicating the calibration mode is displayed as, for example, the lower right of the field of view as the characters “CAL”, and the calibration indices P1 to P5 are described later. In this way, they are displayed blinking sequentially. Of these, the index P1 is displayed at the center of the field of view, the index P2 is displayed at the top of the field of view, the index P3 is displayed at the right side of the field of view, the index P4 is displayed at the bottom of the field of view, and the index P5 is displayed at the left side of the field of view. Is done.

  In each display as described above, normal information display is performed by causing, for example, a G (green) diode in the LED 113 to emit light, and a warning display is performed by, for example, an R (red) diode in the LED 113. It is good to perform by emitting light.

  Next, the principle of parallax correction based on the distance to the subject as described above will be described with reference to FIGS. FIG. 32 is a diagram for explaining the optical relationship between the subject, the photographing optical system, and the CCD, and FIG. 33 is a diagram for explaining the optical relationship between the HOE, the virtual image formed by the HOE, and the eye. FIG. 34 is a diagram for explaining the shift amount of the virtual image necessary for correcting the parallax.

As shown in FIG. 32, the horizontal size of the imaging area of the CCD 87 is h2, the focal length of the imaging optical system 31 is f, the distance from the main point of the imaging optical system 31 to the CCD 87 is f + x, and the main of the imaging optical system 31 is When the distance from the point to the subject O is L2, the horizontal length of the subject O photographed by the CCD 87 is H2, and the horizontal shooting angle of view is θ2, the following relational expression is established. .
[Equation 1]

On the other hand, as shown in FIG. 33, the distance from the position of the photographer's pupil P to the position of the horizontal shooting image frame (virtual image VI) indicating the shooting range is L1, and the horizontal length of the shooting image frame is Assuming that the angle (viewing angle) at which the photographed image frame having the horizontal length H1 from the position of H1 and pupil P is viewed as θ1, the following relational expression is established.
[Equation 2]

In order to shoot in the shooting range set by the photographer, it is necessary that the shooting angle of view is equal to the viewing angle, that is, θ2 = θ1. Under the condition that θ2 = θ1 holds, the right side of Formula 1 and the right side of Formula 2 are equal, so the focal length f of the photographic optical system 31 is obtained as shown in Formula 3 below.
[Equation 3]

On the other hand, the following formula 4 is established from the imaging principle of the lens.
[Equation 4]

From these Formula 3 and Formula 4, the following Formula 5 is derived by erasing x.
[Equation 5]

  From Equation 5, it can be seen that if the subject distance L2 can be obtained, the focal length f can be obtained.

Here, since the relationship of h2 / L2 << H1 / L1 is established in a normal subject, when it is desired to simplify the calculation or when a means for obtaining the subject distance is not provided, an approximate value is obtained by the following formula 6. Can be requested.
[Equation 6]

  Next, the principle of parallax correction will be described with reference to FIG.

  First, as a precondition for explaining the principle of parallax correction, it is assumed that the optical axis direction of the photographing optical system 31 and the direction of the visual axis of the photographer are both perpendicular to the face. It is assumed that the axis and the visual axis are spaced apart by a distance X. Here, the parallax is generated due to the distance X between the optical axis and the visual axis. Note that when the optical axis and the visual axis are relatively inclined, it may cause a large parallax, so that they need to be adjusted to be parallel. For this purpose, the adjustment between the optical axis and the visual axis is performed using the adjustment mechanism as shown in FIGS.

As shown by a solid line and a dotted line in FIG. 34, if the distance to the virtual image VI0 of the photographic image frame indicating the photographic range is the same as the distance to the subject O, the range observed by the photographer The amount of deviation (parallax) from the range captured by the imaging unit 30 is X and is unchanged. However, since the distance L1 from the photographer's pupil P to the virtual image VI1 is actually different from the distance L2 from the main point of the photographing optical system to the subject O, the range indicated by the photographing image frame as a virtual image is actually The parallax correction amount X ′ for matching with the imaging range is as shown in the following Expression 7.
[Equation 7]

When the reciprocal of the magnification of the virtual image VI1 in the photographic image frame (that is, the ratio of the size of the virtual image to the image displayed on the LCD 115) is β, the shift amount of the image displayed on the LCD 115 to correct the parallax. The SP is as shown in the following Expression 8.
[Equation 8]

  Accordingly, the fourth CPU 111 controls the LCD driver 117 so as to shift the position of the image displayed on the LCD 115 by the amount SP expressed by the mathematical formula 8. As a result, the position of the virtual image VI1 is shifted by the distance X ′ to become the virtual image VI2, and the range indicated by the virtual image shooting image frame coincides with the actual imaging range, as indicated by a two-dot chain line in FIG.

  Thus, since the shift amount SP for performing the parallax correction depends on the subject distance L2, it is basically performed whenever the subject distance is different.

を用いて視角Ｓθに換算すると、約１度となって、さほど大きなパララックスが生じているとはいえない。このように、通常の撮影を行う場合には、パララックス補正はほとんど必要がないということができる。その一方で、数式９の右辺括弧内は被写体距離Ｌ２の逆数に比例しているために、視角Ｓθが大きくなって上述した数式８によるパララックス補正が必要となってくるのは、Ｌ２が小さいとき、つまり近接撮影のとき、であることが分かる。 However, for example, in the case of β = 1/100, L1 = 2m, L2 = 2m, and X = 4 cm, the shift amount SP is 0.4 mm. This parallax correction amount is expressed as follows. Formula 9,
[Equation 9]

Is converted to the viewing angle Sθ, it is about 1 degree, and it cannot be said that a very large parallax occurs. In this way, it can be said that parallax correction is hardly necessary in normal shooting. On the other hand, since the right parenthesis in Equation 9 is proportional to the reciprocal of the subject distance L2, the reason why the viewing angle Sθ increases and the parallax correction according to Equation 8 described above is necessary is small. It can be seen that, that is, close-up shooting.

  In the above description, the subject distance L2 is obtained based on the principle of triangulation. That is, the light emitted from the LED for projection 16a is projected onto the subject, the light reflected from the subject is imaged by the CCD 87, and the position of the image on the CCD 87 is obtained based on the principle of triangulation. ing.

  Since L2 is the subject distance in the line-of-sight direction, the LED 16a for light projection is arranged as close as possible to the eye for detecting the line-of-sight, and the line-of-sight direction (θ) is controlled by a known means for controlling the light projection direction. You should make it light up.

  Furthermore, instead of obtaining the subject distance based on the principle of triangulation as described above, the subject distance L2 may be obtained from the position of the focus lens 85 when autofocusing is performed. That is, assuming that the position of the subject is infinite, the high-frequency component of the image signal of the subject within the predetermined range of the same angle of view as the line-of-sight direction (θ) is obtained by the AF processing circuit 98 as shown in FIG. The focus lens 85 may be driven to a position where the maximum is obtained (focus detection is performed by a so-called contrast method), and the subject distance L2 may be obtained in reverse from the position of the focus lens 85. In this case, the subject in the line-of-sight direction (θ) and the subject having the same angle of view as the line-of-sight direction (θ) to be subjected to the AF processing do not exactly match, but the subject has a certain size. There is no practical problem.

  Next, referring to FIG. 35, a technique that takes into account the parallax caused by the imaging unit 30 and the line-of-sight direction detection unit of the line-of-sight direction / angular velocity detection unit 7 being spaced apart by a predetermined distance. explain. FIG. 35 is a diagram illustrating an angle relationship when a subject in the line-of-sight direction is viewed from the imaging unit.

  For example, consider a case where the subject of the imaging optical system 31 of the imaging unit 30 is to be focused on the subject in the direction of the visual line detected by the visual line direction / angular velocity detection unit 7. When the distance to the subject is far, the subject in the direction of the photographer's line of sight as viewed from the imaging unit 30 substantially matches the subject that the photographer is actually gazing at, but when the distance to the subject is close, Deviations occur in these. Therefore, even if the subject at the shifted position is focused, the subject that the photographer is gazing at may not be focused. FIG. 35 shows an example of the positional relationship when such parallax occurs.

Now, it is assumed that the photographer is gazing at a subject O at a distance L in a direction that forms an angle θ with respect to a visual axis parallel to the optical axis of the photographing optical system 31. At this time, let Xe be the distance between the optical axis of the photographic optical system 31 and the pupil of the photographer. Further, an angle formed by the optical axis of the photographing optical system 31 and a straight line connecting the principal point H of the photographing optical system 31 and the subject O is defined as θ ′. Then, the following formula 10 is established.
[Equation 10]

  Accordingly, the line-of-sight direction θ detected by the line-of-sight direction / angular velocity detector 7, the subject distance L detected by the AF processing circuit 98, etc., and Xe determined in terms of design are substituted into the equation 10 to obtain the angle θ 'Is obtained, and the photographing optical system 31 may be focused on the subject corresponding to the direction of the obtained angle θ'. When the subject distance L is large, the second term on the right side of Equation 10 can be ignored, so that θ′≈θ, that is, it is understood that the influence of the parallax need not be considered.

  The subject distance L may be obtained by triangulation as described above, or may be obtained based on the result of focus detection by the contrast method.

  Next, the operation of the camera 1 will be described with reference to FIGS. 54 and 55. FIG. 54 is a flowchart showing a part of the operation of the camera, and FIG. 55 is a flowchart showing another part of the operation of the camera. 54 and 55 show the operation flow of the camera divided into two diagrams for convenience of drawing.

  When the camera 1 is turned on or the system is reset, a photographic image frame indicating a photographing range corresponding to a standard lens (for example, the angle of view is 50 degrees as described above) is displayed on the see-through image. See-through display is performed as shown in FIG. 21 by the display unit 6 (step S1).

  Next, it is determined whether or not a predetermined time has elapsed by checking a timer 123 built in the second CPU 121 (step S2).

  Here, if it is determined that the predetermined time has elapsed, input of various switches such as the first operation switch 162 as shown in FIG. 8 and the second operation switch 171 as shown in FIG. The situation is monitored (step S3).

  The timer in step S2 starts counting again when the predetermined time has passed and the process proceeds to step S3. When the time measured by the timer has not reached the predetermined time, the switch is not monitored in step S3. In this way, by checking the switch input status at predetermined time intervals while checking the timer, it is possible to reduce the load on each CPU and to prevent malfunction due to switch chattering. Yes. Note that the timers in steps S14, S20, and S25, which will be described later, perform the same functions as the timers in step S2.

  When the process of step S3 is completed, a danger prevention subroutine as described later is executed (step S4).

  When the process of step S4 ends or when it is determined in step S2 that the predetermined time has not elapsed, the camera 1 is set to either the auto mode (A) or the manual mode (M). It is determined whether or not these modes are set or a mode other than these modes (ie, full auto mode (FA)) is set (step S5). This mode setting is performed by operating the FA / A / M switch 71 as described above.

  Here, when it is determined that neither the auto mode (A) nor the manual mode (M), that is, the full auto mode (FA) is determined, it is determined whether or not the recording mode is set by the recording switch 74 ( Step S6).

  Here, if the recording mode is not set, the process returns to step S2 and the above-described processing is repeated.

  If it is determined in step S6 that the recording mode is set, the information display 206 (character “FA”) indicating that the recording mode is set and the full auto mode is set, An information display 207 (character “REC”) indicating that recording is in progress is displayed in a see-through manner as shown in FIG. 27 (step S7).

  Then, the moving image data, the angular velocity data, the line-of-sight direction data, and the audio data are recorded in the recording memory 157 as shown in FIG. 43 described later (step S8), and the process returns to the step S2 as described above. Repeat the process.

  On the other hand, if it is determined in step S5 that either the auto mode (A) or the manual mode (M) is set, it is determined whether exposure correction is set (step S9). .

  If it is determined that exposure correction is set, for example, as shown in FIG. 23, the set exposure correction amount (correction value) is displayed as an information display 202 as a see-through display (step S10). .

  When the process of step S10 ends or when it is determined in step S9 that exposure correction has not been set, the view mode (V) in which the camera 1 displays the image captured by the imaging unit 30 in a see-through manner. ) Or the frame mode (F) for displaying only the photographic image frame that is the photographic range is determined (step S11). This mode setting is performed by operating the F / V switch 72 as described above.

  Here, when it is determined that the mode is set to the frame mode (F), the camera 1 is then set to the auto mode (A) or set to the manual mode (M). (Step S12). This mode setting is performed by operating the FA / A / M switch 71 as described above.

  Here, in the auto mode (A), when the focal length of the photographing optical system 31 is equal to or greater than a predetermined value, the image picked up by the image pickup unit 30 is automatically set even if the frame mode (F) is set. It is a mode to enlarge and see-through display. By performing such a display, it is possible to easily confirm the details of the subject without performing troublesome operations even in telephoto shooting, and at the normal focal length (focal length less than the predetermined value), the shooting range. Since only a photographic image frame indicating is displayed, it is possible to shoot without a sense of incongruity even during long-time shooting.

  On the other hand, as described above, the manual mode is a mode in which whether or not to perform the see-through display is manually set, and is usually a mode in which only the photographic image frame is displayed.

  If it is determined in step S12 that the manual mode (M) is selected, the “MANU” information display 205 is performed as shown in FIG. 26 (step S13).

  Then, it is determined whether or not a predetermined time has elapsed by checking the timer (step S14).

  Here, when the predetermined time has elapsed, in the risk prevention process in step S4, based on the distance to the subject obtained in step S41 in FIG. A correction value necessary for correcting the parallax, which is a deviation from the imaging range by the imaging unit 30, is calculated based on the principle as described above (step S15).

  Based on the correction value calculated in step S15, the display of the photographic image frame is updated so that the photographic image frame is displayed at the correct position (step S16). By performing such processing, the shooting range can be accurately displayed even when the subject distance changes.

  On the other hand, if it is determined in step S12 that the auto mode (A) is set, then whether or not the focal length of the photographing optical system 31 is set to the telephoto side larger than the predetermined value α is determined. Is determined (step S17).

  If it is determined that the focal length of the photographic optical system 31 is equal to or less than the predetermined value α, the process proceeds to step S14. When it is determined that the focal length of the photographic optical system 31 is greater than the predetermined value α, or when it is determined in step S11 that the view mode (V) is set, the imaging unit 30 The captured electronic image is superimposed on the subject by the see-through image display unit 6 and displayed in a see-through manner (step S18).

  When the process of step S18 is completed, when the process of step S16 is completed, or when it is determined in step S14 that the predetermined time has not elapsed, the photographer switches the wide switch of the remote control unit 5. It is determined whether or not the size of the photographic image frame has been widened (W) by operating 75b (step S19).

  Here, when it is determined that an operation for widening (W) the size of the photographic image frame is performed, it is determined whether or not a predetermined time has elapsed by checking a timer (step S20).

  If it is determined that the predetermined time has elapsed, the focal length f of the photographing optical system 31 has reached the lower limit value k1 at which the focus can be adjusted. It is determined whether or not the operation is going to be smaller (step S21).

  Here, when it is going to be operated to a value smaller than the lower limit k1 where the focus can be adjusted, as shown in FIG. 28, a warning display 208 is performed by see-through (step S22). Further, when the focal length f has not yet reached the lower limit value k1 at which the focus can be adjusted, the focal length f is reduced by driving the variator lens 82 of the photographing optical system 31, and the photographing range set by the photographer is set. The focal length of the photographic optical system 31 is set so as to become (step S23).

  On the other hand, if it is determined in step S19 that an operation to widen (W) the size of the photographic image frame has not been performed, the size of the photographic image frame is determined via the tele switch 75a of the remote control unit 5. It is determined whether or not an operation for narrowing (T) is being performed (step S24).

  Here, if it is determined that an operation for reducing (T) the size of the photographic image frame is performed, it is determined whether or not a predetermined time has elapsed by checking a timer (step S1). S25).

  If it is determined that the predetermined time has elapsed, the focal length f of the photographing optical system 31 has reached the upper limit value k2 at which the focus can be adjusted. It is determined whether or not the operation is to be made larger (step S26).

  Here, when it is going to be operated to a value larger than the upper limit k2 at which the focus can be adjusted, as shown in FIG. 29, a warning display 209 is displayed by see-through (step S27). If the focal length f has not yet reached the focus adjustable upper limit value k2, the focal length f is increased by driving the variator lens 82 of the photographing optical system 31, and the photographing range set by the photographer is set. The focal length of the photographic optical system 31 is set so as to become (step S28).

  If it is determined in step S24 that an operation for reducing (T) the size of the photographic image frame has not been performed, if it is determined in step S20 or step S25 that a predetermined time has not elapsed, or When the process of step S22, S23, S27 or step S28 is completed, the moving image recording operation (or the first operation switch 162) is performed by the recording switch 74 included in the second operation switch 171 of the remote control unit 5. It is determined whether or not a recording mode has been set by performing a moving image recording operation (step S29).

  Here, when the recording mode is set, as shown in FIG. 25, after the information display 204 with the characters “REC” is performed by see-through (step S30), the recording is started (step S31). .

  When the processing of step S31 ends or when it is determined in step S29 that the recording mode is not set, still image shooting by the release switch 73 included in the second operation switch 171 of the remote control unit 5 is performed. It is determined whether or not the operation (or the still image shooting operation by the first operation switch 162) has been performed (step S32).

  If it is determined that a still image shooting operation has been performed, a still image is first recorded (step S33), and then a still image is recorded as shown in FIG. The information display 210 with the characters “REL” indicating this is performed by see-through (step S34).

  If this step S34 is completed or if it is determined in step S32 that the operation of taking a still image by the release switch 73 has not been performed, the process returns to the process of step S2 and the operation as described above. Repeat this step.

  FIG. 56 is a flowchart showing details of the danger prevention subroutine shown in step S4 of FIG.

  As described above, by operating the switch 39, the display of the see-through image can be prohibited as desired so as to be safe, but instead of this, or together with this, the state of the photographer , And see-through display during the photographing operation is automatically prohibited according to the detected state.

  When this danger prevention process is started, first, the distance to the subject is measured (step S41). The distance measurement principle at this time is as follows.

  The light beam emitted from the light projecting LED 16a of the light projecting light emitting unit 16 disposed at the center of the frame unit 13 is emitted as a substantially parallel light beam by the condenser lens 16b, and is irradiated onto the subject. This light is reflected by the subject and enters the imaging surface of the CCD 87 via the photographing optical system 31.

At this time, the distance between the condenser lens 16b and the photographic optical system 31 is L, the distance from the principal point of the photographic optical system 31 to the imaging surface of the CCD 87 is f1, and the reflected light from the subject is on the imaging surface of the CCD 87. If the distance between the image forming position and the position at which the optical axis of the photographic optical system 31 intersects the imaging surface is ΔL, the distance R to the subject is calculated based on the known triangulation principle as follows: It can be obtained by Expression 11.
[Equation 11]

  The distance R thus obtained is temporarily stored in a RAM 122 provided in the second CPU 121.

  Subsequently, the angular velocity sensors 141 and 142 detect angular velocities about two independent axes and temporarily store them in the RAM 122 (step S42).

  Next, the distance measurement result acquired this time is compared with the distance measurement result acquired previously, and it is determined whether or not an object in front of the photographer is relatively approaching (step S43).

  If it is determined that the object ahead is approaching, it is further determined whether or not the distance L to the subject acquired this time is closer than the predetermined distance k3 (step S44). ).

  Here, when it is determined that the distance L is closer than the predetermined distance k3, a predetermined warning mark is displayed on the see-through image display unit 6 (step S45). Here, only the distance to the front object is considered and the moving speed of the front object is not considered, but in step S43, the distance to the front object is obtained and the moving speed (close The predetermined distance k3 may be set adaptively according to the moving speed. In this case, the predetermined distance k3 is set to be larger as the speed at which the front object approaches is faster. The moving speed of the front object can be measured by detecting the distance L at predetermined time intervals.

  On the other hand, if it is determined in step S43 that the object ahead is not approaching, it is further determined whether or not the photographer is walking based on the angular velocity data (step S46).

  That is, the feature pattern of the output from the angular velocity sensors 141 and 142 attached to the head when a person is walking or when the person is running is stored in the EEPROM 102 in advance. When the output pattern from the angular velocity sensors 141 and 142 matches the characteristic pattern at the time of walking or running, it is determined that the photographer is walking (or running).

  If it is determined in step S46 that the photographer is walking (or running), the process proceeds to step S45 to display a warning as described above.

  Further, when it is determined in step S44 that the distance L to the front object is equal to or greater than the predetermined distance k3, or when it is determined in step S46 that the photographer is not walking (or running). No warning is displayed in step S45, assuming that there is no danger.

  In this way, when the process of step S45 ends or when it is determined that there is no danger in step S44 or step S46, this danger prevention process ends and the process returns to the process shown in FIG. .

  In the above description, warning display is performed. However, instead of this warning display, the see-through display by the see-through image display unit 6 may be forcibly (automatically) turned off.

  In the above description, the pattern of the output values of the angular velocity sensors 141 and 142 is compared with a predetermined feature pattern to detect the photographer's state. However, as a simpler means, the output values of the angular velocity sensors 141 and 142 are It may be possible to display a warning or to prohibit see-through display when the predetermined value continues for a predetermined time or more.

  Furthermore, in the above description, the state where the photographer is walking or running is taken as an example of the state of the photographer. However, the present invention is not limited thereto, and it is possible to cope with a state where it is desirable to prevent danger. good. For example, based on the relative speed with the front object, it is determined whether or not a car or the like is being driven, and even when it is determined that the vehicle is being driven, a predetermined warning or a see-through display is displayed. It is better to ban it. At this time, it is more accurately determined whether it is in a state where it is desirable to prevent danger while driving, or whether it is in a state where it is on a train or bus and there is no need to give a warning or the like. In order to achieve this, the output of the angular velocity sensors 141 and 142 may be used, or other sensors may be provided to further increase the detection accuracy.

  The technique for preventing a danger such as a collision with a front object during photographing is not limited to being applied to a head-mounted camera, but also for a head-mounted display device having a see-through display function. The present invention can be applied, and can also be widely applied to other general cameras and display devices.

  By providing such a risk prevention function, the head mounted camera can be used safely. Such a function is particularly effective in the head-mounted camera according to the present embodiment, which can easily perform shooting while performing a normal action.

  Next, the principle of gaze direction detection will be described with reference to FIGS.

  FIG. 36 is a cross-sectional view of the eye taken along a plane including the optical axis Lm and the visual axis Lm ′ when the optical axis Lm of the illumination light beam coincides with the visual axis Lm ′ in the visual line direction of the eye. In FIG. 36, a large arc centered at the point O is the eyeball 221, an arc centered at the point O ′ is the cornea 222, and a line segment extending from the point where the arc indicating the cornea 222 intersects the arc indicating the eyeball 221. Show the irises 223, respectively.

  When such an eye is irradiated with a parallel light flux having an optical axis of Lm, a light spot Pi (hereinafter, “Purkinje”) is positioned at a substantially midpoint between the center of curvature O ′ of the cornea 222 and the vertex K of the cornea 222. Image ").

  When the optical axis Lm of the light beam that illuminates the cornea 222 and the visual axis Lm ′ that indicates the line-of-sight direction of the eye coincide with each other, as shown in FIG. 36, the pupil that is a circular hole in the iris 223 is displayed. The center, the Purkinje image Pi, the center of curvature O ′ of the cornea 222, and the center of rotation O of the eyeball 221 are on the optical axis Lm. The Purkinje image Pi is generated at the center position of the pupil. At this time, when the eyes are observed from the front, the Purkinje image Pi has a high reflectance, so that it is observed as a bright light image. On the other hand, the pupil and the iris have a significantly low reflectance, so that they are observed darkly.

  FIG. 37 is a diagram in which the output signal of the CCD 134 is plotted in the vertical axis direction when an eye image is captured by the visual line direction / angular velocity detector 7 with the line perpendicular to the optical axis Lm perpendicular to the horizontal axis.

  As illustrated, the image IR of the iris 223 is darker than the image portion of the surrounding eyeball 221, and the pupil image PL is darker than the image IR of the iris 223. On the other hand, the Purkinje image Pi protrudes and is bright. Therefore, it is possible to obtain the position from the center of the iris (or pupil) of the Purkinje image by detecting such a brightness distribution.

  On the other hand, FIG. 38 is a cross-sectional view of the eye by a plane including the optical axis Lm and the visual axis Lm ′ when the optical axis Lm of the illumination light beam intersects the visual axis Lm ′ in the visual line direction. .

  When the eyeball 221 rotates around the rotation center O by the angle θ with respect to the optical axis Lm, a relative shift occurs between the center position of the pupil and the position of the Purkinje image Pi. Now, assuming that the distance from the center of curvature O ′ of the cornea to the center i of the pupil is Lx, the distance d on the imaging surface of the CCD 134 between the pupil center i and the Purkinje image Pi when viewed from the direction of the optical axis Lm (FIG. In FIG. 38, the actual distance is shown, but more accurately, the distance d is assumed to be a distance on the imaging surface).

ここにβ1 は、結像レンズ１３２の結像倍率である。 [Equation 12]

Here, β1 is the imaging magnification of the imaging lens 132.

  FIG. 39 shows the output signal of the CCD 134 when the eye image is captured by the visual line direction / angular velocity detector 7 with the horizontal axis being a line that intersects the optical axis Lm and the visual axis Lm ′ and is perpendicular to the optical axis Lm. It is the figure plotted on the vertical axis | shaft direction.

  As shown in the figure, the Purkinje image Pi is detected at a position shifted from the center of the iris (or pupil) by the distance d.

  Therefore, the image of the eye is formed by the light receiving system of the line-of-sight direction detection unit and imaged, image processing is performed on the captured image data, and the position of the Purkinje image Pi and the center position of the iris (or pupil) are determined. By obtaining the distance d, the line-of-sight direction θ can be obtained. Note that the image formed on the CCD 134 via the light receiving system as shown in FIG. 15 or FIG. 17 is an asymmetrical and distorted image in the vertical and horizontal directions. Such geometric characteristics are preliminarily stored in the EEPROM 102 or the like. After storing the correction information and correcting the distortion based on the correction information, the angle θ indicating the line-of-sight direction is calculated based on the equation 12.

  Next, image processing for obtaining the distance d will be described in detail.

  In the above-described principle, one-dimensional gaze direction detection has been described, but in practice it is necessary to detect a two-dimensional gaze direction. FIG. 40 is a diagram illustrating an eye image obtained from a two-dimensional image sensor. As shown in FIG. 40, the center of the pupil is Op, the Purkinje image is Pi, the predetermined center line passing through the pupil center Op is Lp, and the center line Lp is a straight line connecting the pupil center Op and the Purkinje image Pi. And γ, respectively, the distance between the pupil center Op and the Purkinje image Pi corresponds to the distance d in the equation 12. Therefore, based on this distance d, the line-of-sight direction (tilt angle) θ with reference to the direction of infrared light projection to the viewer's eyes (hereinafter referred to as “reference visual axis”) can be obtained. γ represents the rotation direction of the eyeball. Thus, if the position of the pupil center Op and the position of the Purkinje image Pi in the two-dimensional image formed on the CCD 134 can be detected, the two-dimensional line-of-sight direction represented by θ and γ is obtained. It becomes possible.

  Next, FIG. 57 is a flowchart showing processing for obtaining the position of the pupil center Op and the position of the Purkinje image Pi.

  When this processing is started, first, image data of the observer's eye obtained by photoelectric conversion by the CCD 134 of the line-of-sight direction / angular velocity detection unit 7 is stored in the memory 155 in the control / recording unit 4 (step S51). . Based on this image data, for example, the following processing is performed by the first CPU 161, for example.

  In order to remove noise components due to external light, the CCD 134 first captures image data before projecting infrared light onto the viewer's eyes, and then projects infrared light onto the viewer's eyes. The image data of the CCD 134 when it is illuminated is captured, and the former image data is subtracted from the latter image data to generate image data with improved S / N. Based on this image data, the following is shown. Such processing may be performed.

  In the above description, the image data obtained from the CCD 134 is transferred to the memory 155 in the control / recording unit 4. For example, the image data obtained from the A / D conversion circuit 137 in the line-of-sight direction / angular velocity detection unit 7 is transferred to the subsequent stage. A buffer memory or the like may be provided, the image data may be stored in the buffer memory, and the following processing may be performed by the second CPU 121, for example. With this configuration, since the image data relating to the line-of-sight detection can be immediately processed without being transferred to the control / recording unit 4, the line-of-sight direction can be detected in a shorter time. The follow-up performance when the change is increased.

  Next, a Purkinje image is detected from the acquired image data, and the position of the Purkinje image (for example, an X coordinate and a Y coordinate as shown in FIG. 41. Here, in the example shown in FIG. 41, the center line Lp The X axis is taken in the direction and the Y axis is taken in the direction perpendicular to the center line Lp) (step S52). As described above, since the Purkinje image is extremely brighter than other image portions, for example, a predetermined threshold value THPi is set, and a process of determining an area brighter than the threshold value THPi as a Purkinje image can be employed. . Of course, the processing for determining the Purkinje image is not limited to this, and other image processing can also be employed. As the position of the Purkinje image, the center of gravity of an area brighter than the threshold value THPi may be employed, or the peak position of output data may be employed.

  Subsequently, pixel data of a pixel determined to be a Purkinje image is replaced with pixel data when the Purkinje image is not generated based on pixel data of a pixel in the vicinity of the Purkinje image (step S53). Specifically, the value of pixel data is traced in a two-dimensional direction around the position of the Purkinje image, and an inflection point of the pixel data is detected. Then, assuming that the image inside the closed boundary line formed by connecting the inflection points is the Purkinje image, the image data value of the Purkinje image is replaced with the neighboring image data outside the boundary line. At this time, if there is an inclination of the level of the pixel data between a plurality of neighborhoods related to the Purkinje image, the image data to be replaced with the Purkinje image by correcting the level by performing linear interpolation or the like is used. It is preferable to connect to neighboring image data with a smooth level change.

  Further, as shown in FIG. 41, a predetermined region Sd is set around the Purkinje image Pi (step S54). FIG. 41 is a diagram showing a predetermined area set in the eye image with the Purkinje image as the center. In the example shown in FIG. 41, the predetermined region Sd is a square having one side parallel to the X coordinate and the other side parallel to the Y coordinate. However, the present invention is not limited to this. It may be a circular area. The size of the predetermined region Sd is set so that the entire pupil image is included even when the Purkinje image Pi is displaced to the maximum from the pupil center Op. This setting is performed by designating a length Ld that is half of one side as shown when the region Sd has a square shape as shown in FIG. 41, for example. Further, when the area Sd is circular, the radius is designated.

  Then, the minimum value Vmin in the pixel data included in the area Sd forming a square is detected (step S55).

  Next, a value Vmin + α obtained by adding a predetermined value α to the minimum value Vmin is set. However, this value Vmin + α is set within a range that is lower than the level of the image data in the iris region. Then, the X coordinate and Y coordinate of a pixel having pixel data equal to or smaller than this value Vmin + α are detected (step S56). Note that an area formed by a set of detected pixels is S ′.

  Then, the position of the center of gravity of the area S ′ obtained in step S56 is obtained, and this is set as the position of the pupil center (step S57). Here, specifically, the average value of the X coordinate and the average value of the Y coordinate are obtained for all the pixels included in the region S ′, and this is set as the position of the pupil center.

  In the above description, the center of gravity of the pixel corresponding to the pupil is used to obtain the position of the pupil center. However, the present invention is not limited to this. It is also possible to obtain a point and the midpoint between the maximum position and the minimum position of the Y coordinate and use this as the pupil center. Since the pupil has a substantially circular shape, the image when the eyeball rotates is elliptical. Therefore, it is possible to perform such processing.

  By performing the processing described above, the X and Y coordinates of the Purkinje image and the X and Y coordinates of the pupil center can be obtained.

  The pixels in the above description of the processing may be actual pixels configured on the imaging surface of the CCD 134, or may be virtual pixels that consider a plurality of adjacent pixels as a group. For example, when a plurality of pixels are grouped, there is an advantage that the processing time can be shortened.

  By the way, it can be said that the size of each organ of the eye differs for each age, for each sex, for each race, and for each individual. Also, the relative positional relationship between the eye direction detection unit and the eye depends on the individual difference (individual difference for each device) at the time of manufacturing the eye direction detection unit, the manner in which the head mounting unit 2 is worn, and the like. It changes slightly. Therefore, it cannot be said that the line-of-sight direction obtained by the uniform parameter based on the equation 12 is accurate for all photographers.

  For this reason, the camera 1 according to the present embodiment can perform calibration relating to the detection of the line-of-sight direction as desired. As a result, optimization for each user can be performed.

  FIG. 58 is a flowchart showing a calibration process related to the gaze direction detection.

  For example, when the calibration mode is selected by operating the menu button 63 of the first operation switch 162, the menu selection switches 66, 67, 68, 69, the confirmation switch 65, or the like, the calibration process is started.

  When the process is started, first, which of the calibration indices P1 to P5 as shown in FIG. 31 is displayed is set (step S61).

  Then, the set index is displayed, for example, by blinking (step S62). As described above, this display can be observed by an eye on the side different from the eye on which the line-of-sight direction / angular velocity detection unit 7 including the line-of-sight direction detection unit is provided.

  For example, first, the indicator P1 is blinked, and the observer is prompted to watch the indicator P1. When the observer is in a state of gazing at the index P1, the observer performs a predetermined key operation. When this key operation input is detected, image data is taken in by the CCD 134, and a distance d between the pupil center i and the Purkinje image Pi is calculated based on the equation 12 (step S63). At this time, the value of the distance d obtained by gazing at the index P1 is defined as d1.

  And it is judged whether the process of said step S61-S63 was complete | finished about all the said parameter | index P1-P5 (step S64).

  Here, if the processing has not been completed for all the indexes P1 to P5, the process proceeds to step S61, and the processing as described above is performed for the next index, whereby all the indexes P1 to P5 are obtained. The distances d1 to d5 related to the above are calculated.

として、補正情報Δｄを加えることにする。 Here, the equation 12 is an equation that is established only in an ideal case. In practice, it is necessary to consider various errors as described above. Therefore, by correcting the above equation 12,
[Equation 13]

Then, correction information Δd is added.

  In Equation 13, the angle θ is a value set in advance according to the display positions of the indicators P1 to P5 (corresponding to the indicators P1 to P5, respectively, the angles θ1 to θ5). The distance d is a value actually detected by the line-of-sight direction detection unit (the distances d1 to d5 described above). Furthermore, Lx is assumed to be a different value for each individual.

  By substituting the known angles θ1 to θ5, the distances d1 to d5 obtained in step S63, and the imaging magnification β1 of the imaging lens 132 as described above into this equation 13, Δd And five formulas (primary formulas) with Lx as unknowns. In order to obtain the two unknowns Δd and Lx, it is only necessary to combine two linear expressions including these unknowns. Therefore, 5C2 = 10 (groups) are required as possible solutions. . Then, a predetermined statistical calculation is performed on the set of solutions to obtain Δd and Lx that are considered to be most likely as parameters for each individual (step S65).

  Thereafter, Δd and Lx determined in this way are stored in the EEPROM 102 (step S66), and the calibration process is terminated.

  In actually detecting the direction of the line of sight, Δd and Lx obtained by the calibration as described above and the distance d between the pupil center i and the Purkinje image Pi actually detected are substituted into the above equation 13. Thus, the line-of-sight direction θ is calculated and obtained.

  The line-of-sight data obtained in this way can be used in real time at the time of shooting, and can also be used for image editing as described later after shooting. Therefore, when recording moving image data, as will be described later, the obtained line-of-sight direction is associated with the detected time information and recorded together with the moving image data.

  An example of using the gaze direction data in real time at the time of shooting is to automatically adjust the focus of the shooting optical system 31 of the imaging unit 30 to the subject in the gaze direction. In this case, focus detection may be performed for a predetermined range centered on a portion corresponding to the line-of-sight direction in the subject image formed on the CCD 87. At this time, as described with reference to FIG. 35, the parallax should be considered.

  FIG. 43 is a diagram showing the structure of various data recorded in the process of step S8 of FIG.

  The data recorded in the recording memory 157 has a data structure as shown in FIG. 43, for example.

  That is, the data includes a still image data management area, a moving image data management area, an edited moving image data management area, a still image data area, a moving image data area, an audio data area, an angular velocity data area, and a line-of-sight direction. It has a data area and an edited moving image data area.

  The still image data management area is an area for managing still image data recorded in the still image data area in the auto mode (A) or the manual mode (M). The top address of the data is recorded.

  The moving image data management area is an area for managing moving image data recorded in units in the moving image data area. Details thereof will be described later.

  The edited moving image data management area is an area for managing the edited moving image data area in which the edited moving image data is recorded as will be described later, and details thereof will be described later.

  As described above, the still image data area is an area in which still image data is recorded.

  As described above, the moving image data area is an area in which moving image data is recorded in units.

  The audio data area is an area in which audio data recorded by the first microphone 17 and the second microphone 18 is recorded in units.

  The angular velocity data area is an area where angular velocity data detected by the angular velocity sensors 141 and 142 is recorded in units.

  The line-of-sight direction data area is an area where the line-of-sight direction data calculated by the first CPU 161 and the like based on the image data from the line-of-sight direction / angular velocity detector 7 is recorded in units. Note that the gaze direction data recorded in the gaze direction data area may be the angle θ itself indicating the visual axis direction obtained based on the image data from the gaze direction detection unit of the gaze direction / angular velocity detection unit 7. However, instead of this, the parallax-corrected angle θ ′ that is the angle when the subject in the visual axis direction is viewed from the imaging unit 30 may be recorded, as shown in the above formula 10, or Both of these may be recorded. In this way, in the gaze direction data (gaze direction information), the gaze direction is recorded at a predetermined time interval, so that it is possible to calculate the continuous movement amount in the gaze direction. It is also possible to calculate the moving speed. Therefore, it can be said that the gaze direction data includes at least one of the continuous movement amount in the gaze direction and the movement speed in the gaze direction.

  As described above, the edited moving image data area is an area in which edited moving image data is recorded in units.

  Next, the basic concept of the structure of moving image data will be described.

  The moving image data includes audio data recorded at the same time as the imaging of the moving image data, and angular velocity data and gaze direction data detected at the same time as the imaging of the moving image data. Must be recorded in association with. Therefore, in this embodiment, each of these data is grouped into a unit called a unit every predetermined time, and the start address of the memory area where each data included in the unit is actually recorded is set as the moving image data management area. Are recorded together. Therefore, by referring to the management data recorded in the moving image data management area, it is possible to access moving image data, audio data, angular velocity data, and line-of-sight direction data included in a desired unit.

  For example, it is assumed that the capturing period of moving image data, audio data, and angular velocity data is 1/30 seconds, the capturing period of the line-of-sight direction data is 1 second, and each data for 1 second is collected into one unit. At this time, 30 pieces of image data, audio data, and angular velocity data, and one line-of-sight direction data are recorded in one unit. Accordingly, the moving image data management area includes data for managing 30 pieces of image data, data for managing 30 pieces of audio data, data for managing 30 pieces of angular velocity data, 1 Data for managing the line-of-sight direction data is recorded for one unit. Accordingly, such management data is repeatedly recorded in the moving image data management area by the number of units.

  FIG. 44 is a diagram showing a detailed configuration of the moving image data management area.

  In the moving image data management area, management data related to files is recorded in the number of files starting from the file name.

  As shown in FIG. 44, the management data relating to one file includes a file name, a flag (FLG), a start time, an end time, a unit size, and data relating to the unit. .

  The file name is the name of the file attached to the moving image data.

  The flag (FLG) indicates whether the editing process has been performed.

  The start time indicates the time when shooting of moving image data is started.

  The end time indicates the time when the shooting of the moving image data has ended.

  The unit size indicates a size when moving image data, audio data, angular velocity data, and line-of-sight direction data are recorded in one unit unit.

  Data relating to the above units is provided for the number of units included in this file. For one unit, ADR (A) indicating the start address of the video data and ADR (B) indicating the start address of the audio data are provided for one unit. ), ADR (C) indicating the head address of the angular velocity data, and ADR (D) indicating the head address of the line-of-sight direction data are recorded.

  Accordingly, by referring to this moving image data management area, it is possible to grasp what data is recorded in the recording memory 157 as shown in FIG. 45, for example. FIG. 45 is a diagram illustrating an example of information display obtained by referring to the moving image data management area.

  In the example shown in FIG. 45, a list of file names recorded in the recording memory 157 is displayed, the shooting date and shooting time corresponding to each file, and whether editing processing described later is performed. Is displayed.

  Next, FIG. 46 is a diagram showing a detailed configuration of the edited moving image data management area.

  In this edited moving image data management area, management data relating to files related to the edited moving image data is recorded by the number of files starting from the file name.

  As shown in FIG. 46, the management data relating to one file includes a file name, a start time, an end time, a shooting time, and a recording area.

  The file name, start time, and end time are the same as those recorded in the moving image data management area.

  The shooting time is information indicating the total time of the shot moving image.

  Further, in the recording area, the head address indicating the head address of the recording area is recorded for each unit having a predetermined unit size in the same manner as that recorded in the moving image data management area.

  Next, FIG. 59 is a flowchart showing the editing process.

  For example, when the editing mode is selected by operating the menu button 63 of the first operation switch 162, the menu selection switches 66, 67, 68, 69, the confirmation switch 65, etc., the editing process is started. It has become.

  When this process is started, first, as shown in FIG. 45, the image date (shooting time) and whether or not the image file (that is, the moving image file) that has been continuously shot is already edited as a unit. Is displayed as a list. Then, it waits for the operator to select one or more image files from the displayed images (step S71).

  When the image file is thus selected, the line-of-sight direction data corresponding to the selected image file is obtained by referring to the management information recorded in the moving image data management area as shown in FIG. Read (step S72).

  Then, based on the read gaze direction data, it is determined whether or not the gaze time is equal to or longer than the predetermined time t1 (step S73). Here, whether or not the user is gazing is determined to be gazing at a certain direction if the change in the magnitude of the vector representing the sight line direction is within a predetermined amount.

  If it is determined that the gaze time is equal to or greater than the predetermined time, a zoom-up process is performed (step S74).

  Here, the zoom-up process will be described with reference to FIGS. 47 to 49. 47 is a diagram showing a temporal change in the line-of-sight position in the X direction, FIG. 48 is a diagram showing a temporal change in the line-of-sight position in the Y direction, and FIG. 49 is a diagram showing a temporal change in zooming. In FIGS. 47 to 49, the time plotted on the horizontal axis shows the same time at the same position.

  Note that the X axis and the Y axis are two axes that are orthogonal to each other and describe a vector space indicating the line-of-sight direction.

  Assume that ΔX and ΔY are predetermined line-of-sight position change amounts in the X direction and the Y direction, respectively. At this time, when the line-of-sight position changes in the X direction and the Y direction are ΔX and ΔY or less and the gaze time is the predetermined time t1 or more, as shown in FIG. 49, the image is slowly directed toward the gaze point. Expand. That is, enlargement and trimming (change of the enlarged area) are performed simultaneously so that the gazing point gradually comes to the center of the image. When a certain amount of magnification is reached, the state is maintained for a while and then slowly returned to the original magnification (standard angle of view) toward the end of the gaze time t0.

  In the middle of gaze, the gaze direction may momentarily move elsewhere and return to the original gaze direction again. Considering the gaze time as such gaze movement is noise. Image processing that cannot be included in the image is performed. Since such image processing can be easily executed by known moving average and noise processing, details of the processing are omitted.

  Returning again to the description of FIG. 59, when the zoom-up process in step S74 ends or when it is determined in step S73 that the gaze time is longer than t1, the two directions (yaw direction and pitch direction) are described. Is read (step S75).

  Thereafter, a predetermined editing process is executed on the basis of the read angular velocity data. Before proceeding with the description according to the flowchart of FIG. 59, the basic concept of this editing process will be described with reference to FIGS. Will be described with reference to FIG. FIG. 50 is a diagram showing the time change of the angular velocity, FIG. 51 is a diagram showing the time change of the absolute value of the angular velocity, and FIG. 52 is a diagram showing the change of the time integral value of the angular velocity. 50 to 52 are illustrated such that the times plotted on the horizontal axis indicate the same time at the same position. Also, the angular velocity time integral (ΣωΔt) shown in FIG. 52 corresponds to the amplitude.

  The basic idea of the editing process is as follows. First, the change in the angular velocity ω for a very short time t2 or less is ignored as noise. Next, with a predetermined time t3 (here, t3 is longer than the above t2) as a threshold, when the duration t is less than t3, it is determined that the vibration has a short period (that is, shake). If it is equal to or greater than t3, it is determined that there is a change in the direction that is not shaken. When it is determined that there is a shake, a shake correction is performed if possible, and a cutting process is performed if impossible. Similarly, if it is determined that the change is not in the direction of blurring, if possible, smoothing processing (editing processing that corrects the sudden flow of the image to make a smooth change and extends the recording time) If not possible, the cutting process is performed.

  Based on this concept, the three cases of sections A, B, and C as shown in FIGS. 50 to 52 will be specifically described.

  In section A, as shown in FIG. 51, the duration t is not less than the predetermined time t3 and is considered to be a change in the direction of no shaking, and the absolute value | ω | of the angular velocity can be smoothed. The smoothing process is performed because of the size. As a specific method of the smoothing process, there is a method of inserting a number of frames according to the degree of smoothing between frames (frame images) constituting a moving image. However, with this method, since the level of smoothing increases (stretches the time change of the image), the motion becomes awkward, so a frame that interpolates the motion based on the image data of adjacent frames is created and created. It is better to insert a frame.

  In the section B, the duration t is smaller than the predetermined time t3 and is considered to be shake, and the absolute value of the angular velocity | ω | is large enough to perform shake correction. Perform image stabilization.

  The interval C has a duration t smaller than the predetermined time t3 and is considered to be shaken. However, the absolute value of the angular velocity | ω | is so large that it is not suitable for shake correction. Cutting.

  Although not shown, it is considered that the duration t is equal to or longer than the predetermined time t3 and it is considered that the direction is not shaken. However, when the absolute value of the angular velocity | ω | is not large enough to perform the smoothing process. Also, the image of the section is cut.

  Further, in the time interval before the interval A, the interval between the interval A and the interval B, etc., the absolute value of the angular velocity | ω | is so small that the blur correction is unnecessary and the smoothing process is unnecessary. Because it is small, it is not a subject of editing.

  If cutting is performed based on the above-described concept, for example, an isolated image as shown in FIG. 53 may remain. FIG. 53 is a diagram illustrating a state in which an isolated image remains after cutting, and indicates in binary whether or not an image exists along the time axis. In the example shown in FIG. 53, three sections of section C1, section C2, and section C3 are cut, but image SL1 in section C2 and images SL2 and SL3 in section C3 remain as isolated images. . Such an image is subjected to a removal process as will be described later.

  Returning to the description of FIG. 59 again, the processing after step S76 based on the basic concept as described above will be described.

  It is determined whether or not the duration t of the change in at least one of the angular velocities in the biaxial directions satisfies t2 <t <t3. When this condition is satisfied, the absolute value of the angular velocity | ω It is determined whether or not | satisfies ω1 <| ω | <ω2 (ω1 is the lower limit value of the angular velocity that needs to be corrected and ω2 is the upper limit value of the angular velocity that can be corrected) (step S76).

  Here, when the condition of step S76 is satisfied, blur correction processing is performed by known image processing (step S77).

  If the condition of step S76 is not satisfied, it is determined whether the change time t of the angular velocity ω satisfies t3 ≦ t. If this condition is satisfied, the absolute value of the angular velocity | It is determined whether or not ω | satisfies ω3 <| ω | <ω4 (ω3 is a lower limit value of the angular velocity that requires smoothing and ω4 is an upper limit value of the angular velocity that can be smoothed) (step S78).

  If the condition in step S78 is satisfied, the smoothing process as described above is performed (step S79).

  On the other hand, when the condition of step S78 is not satisfied, a case where the duration t of the change in the angular velocity ω satisfies t2 <t <t3 and the absolute value | ω | of the angular velocity satisfies ω2 ≦ | ω | It is determined whether or not the duration t of the change of ω satisfies t3 ≦ t and the absolute value of the angular velocity | ω | satisfies ω4 ≦ | ω | (step S80).

  If the condition in step S80 is satisfied, the section is cut (step S81).

  Further, when the condition of step S80 is not satisfied, the following three cases are conceivable.

  The first case corresponds to the case where the duration t corresponds to the fluctuation (t2 <t <t3), but the absolute value | ω | of the angular velocity is not more than the lower limit value ω1. At this time, it is not considered that the shake needs to be corrected, and no editing is performed as it is.

  The second case corresponds to a change in the direction in which the duration t is not shaken (t3 ≦ t), but is when the absolute value of the angular velocity | ω | is equal to or less than the lower limit value ω3. At this time, the direction change is considered to be sufficiently gradual and smoothing is unnecessary, and no editing is performed.

  The third case is a case where the duration t of the change in the angular velocity ω is t2 or less (that is, t ≦ t2) and corresponds to noise. This noise can be present in a normal section or in a section that has been subjected to cutting processing (that is, when it remains as an isolated point as shown in FIG. 53). For isolated points, the following processing is performed.

  That is, when any of the processes of step S77, step S79, and step S81 ends, or when the condition of step S80 is not satisfied, isolated points are removed (step S82).

  Thereafter, the edited moving image data is recorded in the recording memory 157 (step S83), and the editing process is terminated.

  In this way, in a case where image quality is considered to deteriorate, correction is performed to improve the image quality within the possible range, and cutting is performed when correction is impossible, so an image with good quality and easy to see can be obtained. Can do. Since the editing process is automatically performed, the user does not need to perform a troublesome operation, which is simple.

  In the example of the editing process as shown in FIG. 59, zooming, cutting, smoothing, and blur correction are performed based on line-of-sight information and angular velocity information as biological information. However, it is not limited to these. For example, based on the temporal variation pattern of the angular velocity data, it is determined whether or not the photographer is walking, and when it is determined that the photographer is walking, zooming processing is performed on the wide angle side than usual. It is possible to do. Conversely, when it is determined that the user is walking, a part of the image is cut out so as to suppress the fluctuation of the image, and the cut out image is zoomed (enlarged) to the telephoto side. good. By performing such processing, it is possible to reproduce a moving image that is visually uncomfortable with less fluctuation of the image.

  In the editing process described above, after shooting, an image is edited based on biological information such as line-of-sight information and head angular velocity information, and the edited image data is recorded on a recording medium. Sometimes it is possible to edit in real time. That is, the contents of a predetermined editing process (hereinafter referred to as “editing information”) and time information regarding the start time of the editing process are stored in the memory in association with the time information from which the original image and the biological information are acquired. Keep it. Then, when reproducing the image data, the original image data and the editing information are simultaneously read out while leaving the original image data, and the original image data is reproduced while being edited in real time based on the editing information (hereinafter referred to as the original image data). Then, this may be referred to as “dynamic editing processing”).

  FIG. 60 is a block diagram showing a main part of the configuration of the dynamic editing processing apparatus.

  This dynamic editing processing apparatus includes an original image memory 231, an operation unit 232, a program correction unit 233, an editing information memory 234, a reading unit 235, an editing processing unit 236, an editing processing program memory 237, a display And an image editing apparatus and an image reproduction apparatus.

  The original image memory 231 is image data recording means, and stores original image data in association with time information relating to the time at which each image is captured (acquired).

  The operation unit 232 is an operation means for manually performing operation input to the dynamic editing processing apparatus from the outside.

  The program correcting unit 233 is editing information correcting means for correcting editing information stored in the editing information memory 234 in response to an operation from the operation unit 232.

  The edit information memory 234 is a biometric information recording means, which will be described later with reference to code information (see FIGS. 61 to 70) related to the edit processing program as described with reference to FIG. 59, and the original image. Editing information related to the original image data stored in the memory 231 (editing information generated based on the biometric information) is stored in association with the time information. If there is biometric information that is not used as editing information, the biometric information may be recorded in the editing information memory 234.

  The reading unit 235 constitutes processing means, and reads the original image data from the original image memory 231 and the edit information from the edit information memory 234 according to the associated time information. The data is transferred to the edit processing unit 236.

  The editing processing unit 236 constitutes processing means, and the original image data read by the reading unit 235 is converted into editing information read by the reading unit 235 (that is, according to the time information). , Editing information associated with the original image data).

  The edit processing program memory 237 constitutes processing means, and stores a specific edit processing program corresponding to an edit pattern as will be described later.

  The display unit 238 is a display means for displaying the image data processed by the editing processing unit 236 so as to be observable.

  Next, details of the information stored in the editing information memory 234 will be described with reference to FIGS.

  61 is a diagram showing a first zooming pattern, FIG. 62 is a diagram showing editing information related to zooming stored in the editing information memory in accordance with the first zooming pattern, and FIG. 63 is a diagram showing the second zooming pattern. FIG. 64 is a diagram showing editing information related to zooming stored in the editing information memory in accordance with the second zooming pattern, FIG. 65 is a diagram showing a third zooming pattern, and FIG. 66 is a third zooming. FIG. 67 shows a fourth zooming pattern, and FIG. 68 shows a fourth zooming pattern stored in the editing information memory in accordance with the fourth zooming pattern. It is a figure which shows the edit information regarding zooming.

  The first zooming pattern as shown in FIG. 61 is a pattern in which zooming is performed from reduction to enlargement during time T1, and thereafter zooming is stopped as it is.

  The editing information memory 234 stores editing information related to zooming as shown in FIG. In this editing information, the lower 4 bits are the zooming pattern, the next 4 bits are the first zooming time (T1), the next 4 bits are the stop time (T2), and the next 4 bits are the first state. The zooming time to return (T3), the most significant 16 bits are time information indicating the start time of editing.

  In this example, the memory capacity for storing each of the times T1, T2, and T3 is 4 bits. However, the present invention is not limited to this, and the capacity may be determined as necessary.

  In the lower 4 bits, a zooming pattern code (here, “0000”) indicating a first zooming pattern is stored.

  In the next 4 bits (T1), information corresponding to the zoom time as shown in FIG. 61 is stored. On the other hand, in this first zooming pattern, since codes T2 and T3 are not defined, the code “0000” is stored in both.

  Next, the second zooming pattern as shown in FIG. 63 is a pattern in which the zooming is performed from the reduction to the enlargement for the time T1, the zooming is stopped for the time T2, and the zooming is performed from the enlargement to the reduction for the time T3. .

  In the second zooming pattern, “0001” is stored as a zooming pattern code as shown in FIG. Each time T1, T2, T3 is also defined and stored.

  Subsequently, the third zooming pattern as shown in FIG. 65 is a pattern in which zooming is performed from enlargement to reduction for the time T1, and then the zooming is stopped as it is.

  In the third zooming pattern, as shown in FIG. 66, “0010” is stored as the zooming pattern code. Further, only the time T1 is defined and stored, and the code “0000” indicating that it is not defined is stored at the times T2 and T3 as described above.

  The fourth zooming pattern as shown in FIG. 67 is a pattern in which zooming is performed from enlargement to reduction for time T1, stop zooming for time T2, and zoom from reduction to enlargement for time T3.

  In the fourth zooming pattern, as shown in FIG. 68, “0011” is stored as the zooming pattern code. Each time T1, T2, T3 is also defined and stored.

  FIG. 69 is a diagram illustrating an example of a code representing smoothing editing information.

  In the lower 4 bits, a code “0100” indicating that the editing process is smoothing is performed. In the next 8 bits, a time interval from the start to the end of smoothing (smoothing time) is displayed. In the upper 16 bits, Time information representing the start time of editing is stored.

  FIG. 70 is a diagram illustrating an example of a code representing cutting editing information.

  The lower 4 bits are a code “0101” indicating that the editing process is cutting, the next 8 bits are the time interval from the start to the end of cutting, and the most significant 16 bits are the editing start time. Each time information representing is stored.

  Therefore, the reading unit 235 reads such editing information stored in the editing information memory 234 in association with the time information in synchronization with the image data stored in the original image memory 231. .

  The edit processing program stored in the edit processing program memory 237 is a program for executing an edit pattern corresponding to the least significant 4 bits of the edit information stored in the edit information memory 234.

  Then, the editing processing unit 236 reads the editing pattern from the editing information read by the reading unit 235, reads an editing processing program corresponding to the editing pattern from the editing processing program memory 237, and performs predetermined editing processing. Execute.

  The edited image is displayed on the display unit 238.

  The dynamic editing process as described above is automatically performed. However, since it is convenient if the user can perform editing processing according to individual preference, in this embodiment, as shown in FIG. 60, an operation unit 232 and a program correction unit 233 are provided. Thus, the editing information stored in the editing information memory 234 can be manually corrected from the outside.

  That is, when editing the editing information, a known function related to playback, that is, primary stop, return, slow playback, etc. of playback is used to observe the image on the display unit 238 while FIG. 62, FIG. The editing information as shown in FIGS. 66, 68, 69, and 70 is changed. At this time, the editing result can be confirmed by the display unit 238, and since the original image is stored as described above, the editing operation can be performed any number of times. It is possible to obtain an edited image that best suits the user's preference.

  In addition, since the detection of the gaze direction as described above requires a high level of technology and high cost, it is cost effective to provide a gaze direction detection device for use only in the editing process described above. I can't say that. Therefore, if only the editing process as described above is used, even if the control is performed using the angular velocity information instead of the line-of-sight information, the cost performance can be improved while obtaining substantially the same effect. . In this case, based on the angular velocity information, when the change in the angle of the head is within a certain range for a predetermined time or more, it is assumed that the observer is observing almost one direction of the subject, as shown in FIG. It is conceivable to perform the same processing as the processing based on accurate line-of-sight information (for example, the zoom-up processing in step S74).

  When the dynamic editing processing apparatus as shown in FIGS. 60 to 70 is used, a recording medium for storing the image data after the editing process becomes unnecessary, and particularly when recording moving image data. A large memory saving effect can be obtained. In addition, since the user can change the editing information for performing the automatic editing process according to his / her preference, the editing process reflecting the intention of each individual can be performed.

  Of the processes shown in FIGS. 54 to 59, the process related to imaging and the process related to line-of-sight direction detection are mainly performed by the second CPU 121. Further, the fourth CPU 111 mainly performs processing relating to see-through image display and light projection processing for distance measurement. Further, the AF processing and AE processing are mainly performed by the third CPU 103. The first CPU 161 mainly performs image data recording processing, editing processing, reproduction processing, and operation switch detection processing. As described above, the CPU to which the peripheral device having each related function is connected is configured to mainly execute processing related to the function. However, since the CPUs are connected to each other, the processing may be appropriately distributed according to the processing capability.

  In the above description, the HOE 25 is used as the combiner of the see-through image display unit 6, but instead, a convex lens optical system, a concave lens optical system, a half mirror, a free-form surface optical system made of glass, plastic, or the like. It is also possible to use a combination of these.

  Further, in the above description, the LED 113, the condenser lens 114, and the LCD 115 are used as means for projecting an image in the see-through image display unit 6, but instead of these, an EL (Electro Luminescence) panel, a self-luminous plasma display, or the like is used. Other display elements may be used.

  According to the first embodiment, biological information related to the movement of the photographer himself / herself is recorded, and a predetermined editing process is subsequently performed on the recorded image data based on the recorded biological information. Since the camera can be configured, the configuration of the camera can be simplified, and the photographer can perform shooting without having to concentrate on the shooting operation. Therefore, the photographer can take the same natural action as a person other than the photographer without feeling the burden of photographing. Since the image editing is performed based on the photographer's biological information, it is possible to automatically perform image editing in accordance with the photographer's intention. When editing image data after recording, it is relatively easy to edit because it eliminates the need for complicated processing such as prediction processing, which is necessary when editing in real time during shooting. Processing can be performed. In addition, since the edited image can be displayed, it is possible to enjoy an image that is easy to view.

  In addition, since the imaging unit and the line-of-sight direction / angular velocity detection unit can be mounted on the photographer's head, the subject that the photographer is observing can be detected with high accuracy.

  Furthermore, since the rotation of the photographer's head itself is detected based on the angular velocity, editing processing such as correction of image blur due to head shake, cutting, and extended recording time is automatically performed. Is possible.

  Since the gaze direction, which is the direction of the photographer's gazing point, is detected, editing processing such as electronic zooming (enlargement / reduction) and trimming can be automatically performed. At this time, since the line-of-sight direction information includes at least one of the movement speed in the line-of-sight direction and the continuous movement amount in the line-of-sight direction, the will of the photographer can be reflected more faithfully.

  Further, in consideration of the parallax between the line-of-sight direction / angular velocity detection unit and the imaging unit, an angle θ ′ viewed from the imaging unit corresponding to the line-of-sight direction θ detected by the line-of-sight direction / angular velocity detection unit is obtained. Therefore, even when the subject is at a short distance, it is possible to accurately grasp the imaging range (the photographer's point of sight in the captured image) by the imaging unit corresponding to the photographer's gaze direction. Editing processing based on can be performed accurately.

  Furthermore, a head-mounted display device that allows observation while performing natural behavior tends to be used, for example, while walking or driving a car. If it is used in such a state, the display of shooting information displayed on the display means may be distracted, and there is a possibility of colliding with a front object. Since the state is detected and a warning is given or display is prohibited at the time of detection, this can be prevented beforehand.

  In addition, when performing line-of-sight detection, corneal reflected light that is extremely brighter than other regions is used, so that the approximate position of the pupil can be easily obtained. At this time, since an LED that emits infrared rays is used as illumination for obtaining corneal reflected light, observation of the subject by the observer is not hindered.

  Further, the pixel having the minimum value is detected from the region of the predetermined range including the position of the corneal reflection light, and a value obtained by adding the predetermined value to the minimum value (this is larger than the minimum pixel value, Since a pixel group having a pixel value smaller than the threshold value is determined as a pixel group belonging to the pupil image, the pupil region can be easily and It can be obtained with high accuracy.

  Since the center of the pupil region is the center of the pupil, even if some noise is included in the pixel data, the position of the pupil center can be obtained accurately and accurately.

  In addition, the line-of-sight direction can be easily and accurately obtained based on the position of the pupil center obtained as described above and the position of the corneal reflected light.

  On the other hand, when the original image data before the editing process is read from the recording medium and the image after the editing process is displayed while executing the editing process based on the editing information based on the biometric information, the image after the editing process is displayed. A recording medium for storing data becomes unnecessary, and in particular, the memory capacity of the recording medium for recording moving image data can be greatly saved.

  At this time, the editing information for performing the automatic editing process can be changed by the user according to his / her preference, so that the editing process reflecting the intention of each individual can be performed.

  In addition, since the diopter adjusting lenses are arranged on the front surface of the transparent optical member, even observers with different visual acuity can observe substantially directly (that is, in this case, the diopter adjusting lens). It is possible to superimpose a predetermined image on the subject.

  In addition, it is possible to construct an eyeglass-type camera with a simple structure and a beautiful and natural design.

  Since the diopter adjustment lens can be easily detached separately from the transparent optical member, diopter adjustment according to the user can be easily performed for each display device. In addition, at this time, even if the visual acuity of the left eye and the right eye is different, it is possible to wear a lens suitable for each eye.

  In addition, since the transparent optical member and the imaging unit are integrally held by the frame unit, even if the diopter adjustment lens is replaced, the angle adjustment between the transparent optical member and the imaging unit can be performed. This is a user-friendly head mounted camera with a diopter adjustment lens.

  In addition, this invention is not limited to the Example mentioned above, Of course, a various deformation | transformation and application are possible within the range which does not deviate from the main point of invention.

  The present invention can be suitably used for an image editing apparatus, an image editing method, an image editing program, and a recording medium for recording an image editing program for performing editing processing on image data.

1 is a perspective view showing a usage pattern of a head-mounted camera in Embodiment 1 of the present invention. The front view which shows the head mounting part in the said Example 1. FIG. The top view which shows the head mounting part in the said Example 1. FIG. The right view which shows the head mounting part in the said Example 1. FIG. In the said Example 1, the top view which shows the control / recording part of the state which closed the operation panel. In the said Example 1, the right view which shows the control / recording part of the state which closed the operation panel. In the said Example 1, the left view which shows the control / recording part of the state which closed the operation panel. In the said Example 1, the top view which shows the operation switches arrange | positioned at the operation panel. In the said Example 1, the perspective view which shows the control / recording part of the state which opened the operation panel. The top view which shows the structure of the remote control part in the said Example 1. FIG. FIG. 3 is a block diagram illustrating a configuration mainly related to an electronic circuit of the head-mounted camera in the first embodiment. FIG. 3 is a diagram for explaining the principle of the optical system of the see-through image display unit in the first embodiment. FIG. 3 is a front view including a partial cross section illustrating a configuration of an optical system of a see-through image display unit in the first embodiment. FIG. 3 is a left side view illustrating the configuration of the optical system of the see-through image display unit in the first embodiment. FIG. 3 is a front view including a partial cross section showing a configuration example of an optical system of a line-of-sight direction detection unit in the first embodiment. FIG. 4 is a left side view illustrating a configuration example of an optical system of a line-of-sight direction detection unit in the first embodiment. The front view including the partial cross section which shows the other structural example of the optical system of the gaze direction detection part in the said Example 1. FIG. FIG. 7 is a left side view illustrating another configuration example of the optical system of the line-of-sight direction detection unit in the first embodiment. In the said Example 1, the top view and right view which show the structure which attaches an imaging part to a frame part. In the said Example 1, the right view which shows the structure of the hole provided in the flame | frame part in order to attach an imaging part. The figure which shows the example of a display of the initial state in the see-through display of the said Example 1. FIG. In the said Example 1, the figure which shows the example of a display when zoom to tele is performed. The figure which shows the example of a display when zoom to wide and exposure correction | amendment are performed in the said Example 1. FIG. The figure which shows the example of a display when performing electronic view display in the said Example 1. FIG. The figure which shows the example of a display during video recording in the said Example 1. FIG. In the said Example 1, the figure which shows the example of a display at the time of manual mode. The figure which shows the example of a display during video recording in the full auto mode in the said Example 1. FIG. FIG. 6 is a diagram illustrating an example of a warning display that is displayed in a see-through manner when an attempt is made to operate the photographing optical system to a smaller one in spite of the fact that the focal length of the photographing optical system has reached a lower limit value that allows focus adjustment. . FIG. 6 is a diagram illustrating an example of a warning display that is displayed in a see-through manner when an attempt is made to operate the photographic optical system in a larger direction even though the focal length of the photographing optical system has reached an upper limit value that allows focus adjustment. . FIG. 6 is a diagram illustrating a display example when an operation for capturing a still image is performed in the first embodiment. FIG. 6 is a diagram illustrating a display example in a calibration mode according to the first embodiment. FIG. 3 is a diagram for explaining an optical relationship among a subject, a photographing optical system, and a CCD in the first embodiment. In the said Example 1, the figure for demonstrating the optical relationship between HOE, the virtual image formed by this HOE, and eyes. FIG. 6 is a diagram for explaining a virtual image shift amount necessary to correct parallax in the first embodiment. FIG. 4 is a diagram illustrating an angle relationship when a subject in the line-of-sight direction is viewed from the imaging unit in the first embodiment. FIG. 6 is a cross-sectional view of the eye by a plane including the optical axis Lm and the visual axis Lm ′ when the optical axis Lm of the illumination light beam and the visual axis Lm ′ in the eye sight line direction coincide with each other in the first embodiment. FIG. 4 is a diagram in which CCD output signals are plotted in the vertical axis direction when an eye image is picked up by the line-of-sight direction / angular velocity detection unit with the line perpendicular to the optical axis Lm perpendicular to the horizontal axis in the first embodiment. FIG. 3 is a cross-sectional view of the eye by a plane including the optical axis Lm and the visual axis Lm ′ when the optical axis Lm of the illumination light beam intersects with the visual axis Lm ′ in the eye viewing direction in the first embodiment. In the first embodiment, the CCD output signal when the eye image is captured by the visual line direction / angular velocity detector with the line intersecting the optical axis Lm and the visual axis Lm ′ and perpendicular to the optical axis Lm as the horizontal axis. The figure plotted in the vertical axis direction. FIG. 3 is a diagram showing an eye image obtained from a two-dimensional image sensor in the first embodiment. In the said Example 1, the figure which shows the predetermined area | region set to the image of an eye centering on a Purkinje image. In the said Example 1, the figure which shows the time-sequential structure of the signal output from a multiplexing circuit. The figure which shows the structure of the various data recorded in the process of step S8 of the said FIG. In the said Example 1, the figure which shows the detailed structure of a moving image data management area. The figure which shows an example of the information display obtained by referring the moving image data management area in the said Example 1. FIG. In the said Example 1, the figure which shows the detailed structure of the edited moving image data management area. In the said Example 1, the figure which shows the time change of the gaze position of a X direction. In the said Example 1, the figure which shows the time change of the eyes | visual_axis position of a Y direction. The figure which shows the time change of the zooming in the said Example 1. FIG. The diagram which shows the time change of the angular velocity in the said Example 1. FIG. The diagram which shows the time change of the absolute value of angular velocity in the said Example 1. FIG. The diagram which shows the mode of the change of the time integral value of the angular velocity in the said Example 1. FIG. In the said Example 1, the figure which shows a mode that the isolated image remains after performing cutting. 7 is a flowchart showing a part of the operation of the camera in the first embodiment. 9 is a flowchart showing another part of the operation of the camera in the first embodiment. The flowchart which shows the detail of the danger prevention subroutine shown to step S4 of the said FIG. 7 is a flowchart showing processing for obtaining the position of the pupil center and the position of the Purkinje image in the first embodiment. 7 is a flowchart illustrating calibration processing according to gaze direction detection in the first embodiment. 7 is a flowchart showing editing processing in the first embodiment. FIG. 3 is a block diagram illustrating a main part of the configuration of the dynamic editing processing apparatus according to the first embodiment. FIG. 3 is a diagram illustrating a first zooming pattern in the first embodiment. The figure which shows the edit information regarding the zooming memorize | stored in the edit information memory according to the 1st zooming pattern in the said Example 1. FIG. The figure which shows the pattern of the 2nd zooming in the said Example 1. FIG. The figure which shows the edit information regarding the zooming memorize | stored in the edit information memory according to the 2nd zooming pattern in the said Example 1. FIG. The figure which shows the pattern of the 3rd zooming in the said Example 1. FIG. The figure which shows the edit information regarding the zooming memorize | stored in the edit information memory according to the 3rd zooming pattern in the said Example 1. FIG. The figure which shows the pattern of the 4th zooming in the said Example 1. FIG. The figure which shows the edit information regarding the zooming memorize | stored in the edit information memory according to the 4th zooming pattern in the said Example 1. FIG. In the said Example 1, the figure which shows the example of the code | cord | chord showing the edit information of smoothing. In the said Example 1, the figure which shows the example of the code | symbol showing the edit information of cutting.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Head-mounted camera 2 ... Head mounting part 3 ... Cable 4 ... Control / recording part (Image data recording means, biological information recording means, processing means, parallax correction means, calculation means, danger prevention means, relative speed Detection means, relative speed distance detection means, walking detection means, corneal reflected light detection means, region setting means, minimum value detection means, pupil detection means, eye direction calculation means)
5. Remote control part (remote control part)
6 ... See-through image display section (display means, danger prevention means)
7: Gaze direction / angular velocity detection unit (biological information detection means, gaze direction detection means, angular velocity detection means)
DESCRIPTION OF SYMBOLS 11 ... Front part 12 ... Temple part 13 ... Frame part 14, 15 ... Transparent optical member 16 ... Light emission part for light projection (ranging means)
16a ... LED for floodlight
16b ... Condensing lens 17 ... First microphone 18 ... Second microphone 19 ... Nose pad 20 ... Rim (mounting means)
21, 22 ... Diopter adjustment lens 24 ... Holographic optical element (HOE, infrared light projection means)
25. Holographic optical element (HOE)
26: Cell modern 27: Storage unit 30: Imaging unit (imaging means, ranging means)
31 ... Shooting optical system 32 ... Cable connection terminal 33 ... Base (adjustment means)
34, 35 ... Screw (pitch direction adjusting means)
36, 37 ... screws (yaw direction adjusting means)
39 ... Switch (Danger prevention operation means)
41 ... Control / Recording main body 42 ... Operation panel 44 ... Power switch 48 ... LCD display element (display means)
49 ... Cable connection terminal 50 ... AV / S connection terminal 51 ... PC connection terminal 54 ... Recording memory insertion port 55 ... Battery insertion port 56 ... Speaker 59 ... Play / Stop switch 61, 62 ... Switch 63 ... Menu button 65 ... Confirm Switch 66, 67, 68, 69 ... Menu selection switch 71 ... FA / A / M switch (switching means)
72 ... F / V switch (switching means)
73 ... Release switch 74 ... Recording switch 75 ... Zoom switch (photographing frame setting means)
76 ... Exposure compensation switch 78, 79 ... Hinge 87 ... CCD (imaging device)
88 ... CDS / AGC circuit (signal processing means)
89... A / D conversion circuit (signal processing means)
90 ... Timing generator (TG)
91 ... CCD driver 92, 93, 94 ... Ultrasonic motor (USM)
95 ... USM driver 96 ... Aperture shutter driver 97 ... Auto exposure processing circuit (AE processing circuit)
98 ... Autofocus processing circuit (AF processing circuit)
102 ... EEPROM
103 ... 3rd CPU
104: Multiplexing circuit 111: Fourth CPU
112 ... LED driver 113 ... LED (projection means)
114 ... Condensing lens (projection means)
115 ... LCD display element (projection means, display element)
116: Holographic optical element (HOE)
117 ... LCD driver (correction means, viewing angle adjustment means)
118 ... LED driver 121 ... 2nd CPU (control means)
122 ... RAM
123 ... Timer 124 ... LED driver (infrared light projection means)
125 ... LED (infrared light projection means)
126 ... Condensing lens (infrared light projection means)
127: Reflection mirror (infrared light projection means)
128 ... half mirror 129 ... holographic optical element (HOE)
131: Reflecting mirror 132 ... Imaging lens 133 ... Band pass filter 134 ... CCD (two-dimensional photoelectric conversion means)
135 ... CCD driver 136 ... CDS / AGC circuit 137 ... A / D conversion circuit 138 ... TG
141, 142 ... Angular velocity sensor (angular velocity detecting means)
151 ... Separation circuit 152 ... DSP circuit 153 ... Delay circuit 154 ... Multiplexing circuit 155 ... Memory 156 ... Compression / decompression circuit 157 ... Memory for recording (image data recording means, biological information recording means)
158 ... D / A conversion circuit 160 ... LCD driver 161 ... first CPU (control means)
162 ... 1st operation switch 163 ... reception circuit 164 ... power supply circuit 171 ... 2nd operation switch 172 ... decoder 173 ... transmission circuit 174 ... power supply circuit 191 ... hole (pitch direction adjusting means)
192 ... long hole (pitch direction adjusting means)
193 ... hole (yaw direction adjusting means)
194 ... slot (yaw direction adjusting means)
201 ... Shooting image frame 203 ... Electronic images 202, 204, 205, 206, 207, 210, 211 ... Information display 208, 209 ... Warning display 231 ... Original image memory (image data recording means)
232 ... Operation unit (operation means)
233 ... Program correction section (editing information correction means)
234 ... Editing information memory (biological information recording means)
235 ... Reading unit (processing means)
236 ... Editing processing section (processing means)
237 ... Editing processing program memory (processing means)
238 ... Display section (display means)
Agent Patent Attorney Susumu Ito

Claims (11)

Biometric information recording means for recording biometric information related to the movement of the photographer when observing the subject in association with time information relating to the time when the biometric information was recorded;
Image data recording means for recording image data in association with the time information;
Processing means for reading the image data recorded by the image data recording means, and performing a predetermined editing process on the image data based on the biometric information recorded at the same time as the read image data;
An image editing apparatus comprising:   The image editing apparatus according to claim 1, wherein the image data is obtained by photographing with a camera mounted on a photographer's head.   The image editing apparatus according to claim 1, wherein the biological information is angular velocity information related to an angular velocity of a photographer's head.   Based on the angular velocity information, recorded image data is configured to perform at least one of cutting, blurring correction, recording time extension, and electronic zooming as the editing process. The image editing apparatus according to claim 3, wherein:   The image editing apparatus according to claim 1, wherein the biological information is gaze direction information related to a gaze direction of a photographer.   The image editing apparatus according to claim 5, wherein electronic zooming of recorded image data is performed as the editing process based on the line-of-sight direction information.   The image editing apparatus according to claim 5, wherein the line-of-sight direction information is information including at least one of a movement speed in the line-of-sight direction and a continuous movement amount in the line-of-sight direction.   The image editing apparatus according to claim 1, wherein the information on the editing process is configured to be corrected by a manual operation. Recording biometric information and image data related to the movement of the photographer when observing the subject in association with time information related to the time when the biometric information and image data were recorded;
Reading the recorded image data and performing a predetermined editing process on the image data based on the biometric information recorded at the same time as the read image data;
An image editing method comprising: Recording biometric information and image data related to the movement of the photographer when observing the subject in association with time information related to the time when the biometric information and image data were recorded;
Reading the recorded image data and performing a predetermined editing process on the image data based on the biometric information recorded at the same time as the read image data;
An image editing program for causing a computer to execute. Recording biometric information and image data related to the movement of the photographer when observing the subject in association with time information related to the time when the biometric information and image data were recorded;
Reading the recorded image data and performing a predetermined editing process on the image data based on the biometric information recorded at the same time as the read image data;
A recording medium for recording an image editing program for causing a computer to execute.

Images