JP2022140328A - Image pickup apparatus, portable device, calibrator, control method therefor, and program - Google Patents

Abstract

To provide an image pickup apparatus, a portable device, a calibrator, a control method therefor, and a program, capable of eliminating manual change of an image pickup direction during picking up an image and capable of easily obtaining an image that records experience even while focusing attention on the experience.SOLUTION: A camera body 1 includes: a face direction detection unit 20 that is worn on a body other than a head of a user and detects an observation direction of the user; an image-capturing unit 40 that is worn on the body of the user and picks up an image; and a transmission unit 70 that outputs an image corresponding to the observation direction among the images picked up by the image-capturing unit 40.SELECTED DRAWING: Figure 4

Description

The present invention relates to an imaging device, a mobile device, a calibrator, and control methods and programs thereof, and more particularly to an imaging device, a mobile device, a calibrator used as an action camera, and control methods and programs thereof.

Traditionally, when taking pictures with a camera, the photographer had to keep pointing the camera toward the subject being photographed. I couldn't concentrate on the experience of being there because I was forced to concentrate.

For example, in terms of imaging operations, the parent, who is the photographer, cannot play with the child while the child is being imaged, and if the child tries to play with the child, the image cannot be captured. occurs.

In addition, in terms of concentrating on shooting, when shooting while watching sports, the photographer may not be able to support or remember the game content, and when focusing on watching sports, shooting may not be possible. problems arise. Similarly, when taking pictures during a group trip, the photographer cannot experience the same level of excitement as the other members, and giving priority to the experience causes a problem that the taking of pictures is neglected.

Therefore, Patent Document 1 discloses a technique of using a second camera that captures an image of a user in addition to a first camera that captures an image of a subject. In this technology, the user's moving direction and line-of-sight direction are calculated from the image captured by the second camera, the imaging direction of the first camera is determined, and the subject is estimated and captured from the user's preference and state. .

Further, in Patent Document 2, a sensor including a gyro and an acceleration sensor is mounted on the head to detect the observation direction of the photographer (user), and the sensor is detected from an imaging device separately mounted on the body or bag. An image recording system is disclosed that captures a detected viewing direction.

JP 2007-74033 A JP-A-2017-60078

However, in Patent Document 1, the second camera captures an image of the user from a position distant from the user. High optical performance is required for the second camera. In addition, there is a problem that the image processing of the image captured by the second camera requires a high arithmetic processing ability, and the apparatus becomes large-scale. Furthermore, even with that, the user's observation direction cannot be calculated precisely, so the subject cannot be accurately estimated from the user's preference and condition, and the image is different from the image desired by the user. There was a problem that the image was captured.

In addition, in Patent Document 2, since the observation direction of the user is directly detected, the user must wear a sensor on the head, and the user must wear some device on the head as described above. I can't solve the hassle of the time. In addition, when the sensor consists of a gyro sensor or an accelerometer, a certain degree of accuracy can be obtained in detecting the relative observation direction, but the accuracy of detecting the absolute observation direction, especially in the horizontal rotation direction, cannot be obtained. had a problem.

Therefore, an object of the present invention is to eliminate the need to change the imaging direction by hand during imaging, and to easily acquire a video recording the experience while concentrating on the experience on the spot. It is to provide a calibrator, a control method thereof, and a program.

An imaging device according to claim 1 of the present invention is mounted on a user's body other than the head, and includes observation direction detection means for detecting the user's observation direction, and is mounted on the user's body, It is characterized by comprising an imaging means for imaging an image, and an image output means for outputting an image corresponding to the viewing direction based on the image taken by the imaging means.

A seventeenth aspect of the present invention is a portable device wirelessly connected to the imaging device, comprising video file receiving means for receiving a video file generated by the imaging device, and extracting metadata from the video file. a first extracting means for extracting; a second extracting means for extracting a video of a frame encoded together with the extracted metadata from the video file; and using the metadata extracted by the first extracting means. frame image correcting means for correcting the image of the frame extracted by the second extracting means; and moving image recording means for recording the image of the frame corrected by the frame image correcting means as a moving image. It is characterized by

A calibrator according to claim 19 of the present invention is a calibrator wirelessly connected to the imaging device, comprising first display means for displaying a positioning index, face detection means for detecting a face, and a second display means for displaying a button for transmitting the calibration instruction to the imaging device when it is determined from the detection result that the user is looking at the positioning index, the button being pressed by the user. It is characterized by

ADVANTAGE OF THE INVENTION According to this invention, it becomes unnecessary to change the imaging direction with a hand during imaging, and the image|video which recorded the experience can be acquired simply, while concentrating on the experience on the spot.

1 is an external view of a camera body including an imaging/detection unit as an imaging device according to Example 1. FIG. It is a figure which shows a mode that the user hung the camera main body. It is the figure which looked at the battery part in the camera main body from the back of FIG. 1A. 1 is an external view of a display device as a mobile device according to Example 1, which is configured separately from a camera body; FIG. It is the figure which looked at the imaging|photography / detection part in a camera main body from the front. FIG. 4 is a diagram showing the shape of the band portion of the connecting portion in the camera body; It is the figure which looked at the imaging|photography / detection part from the back side. It is the figure which looked at the imaging|photography / detection part from the top. FIG. 4 is a diagram showing the configuration of a face direction detection section arranged inside the photographing/detection section and below a face direction detection window in the camera body; It is the figure which looked at the state which the user hung the camera main body from the user's left side. It is a figure explaining the detail of a battery part. 3 is a functional block diagram of a camera body according to Example 1. FIG. 3 is a block diagram showing the hardware configuration of the camera body; FIG. It is a block diagram which shows the hardware constitutions of a display apparatus. 4 is a flow chart showing an outline of image recording processing according to the first embodiment, which is executed in a camera body and a display device; FIG. 7B is a flowchart of a subroutine of a preparatory operation process in step S100 of FIG. 7A according to the first embodiment; FIG. 7B is a flowchart of a subroutine of face direction detection processing in step S200 of FIG. 7A according to the first embodiment; 7B is a flowchart of a subroutine of a recording direction/range determination process in step S300 of FIG. 7A according to the first embodiment; 7B is a flowchart of a subroutine of recording area development processing in step S500 of FIG. 7A according to the first embodiment; FIG. 7B is a diagram for explaining the processing from steps S200 to S600 in FIG. 7A in moving image mode; FIG. 10 is a diagram showing an image of a user seen through a face direction detection window; FIG. 10 is a diagram showing a case where an indoor fluorescent lamp is reflected as a background in an image of the user seen through the face direction detection window; When the user and the fluorescent lamp as the background shown in FIG. 8B are imaged on the sensor of the infrared detection processing device through the face direction detection window in a state where the infrared LED of the infrared detection processing device is not lit. is a diagram showing an image of . FIG. 8B is a diagram showing an image when the user shown in FIG. 8B and a fluorescent lamp as its background are imaged on the sensor of the infrared detection processing device through the face direction detection window with the infrared LED turned on. is. FIG. 8C is a diagram showing a difference image calculated by subtracting the image of FIG. 8C from the image of FIG. 8D; FIG. 8E is a diagram showing a result of adjusting the gradation of the difference image in FIG. 8E by adjusting the scale to the light intensity of the reflected infrared rays projected onto the user's face and neck. FIG. 8F is a diagram in which symbols indicating each part of the user's body, double circles indicating the neck position, and black circles indicating the chin position are superimposed on FIG. 8F. FIG. 8E shows a difference image calculated in the same manner as in FIG. 8E when the user's face is facing right; The shading of the difference image in FIG. 8H is adjusted according to the light intensity of the reflected infrared rays projected on the user's face/neck, and the double circle indicating the neck position and the black circle indicating the chin position are obtained. is a diagram in which the symbols of are superimposed. FIG. 10 is a diagram showing an image of a user seen through the face direction detection window when the user faces 33° above the horizontal; When the user faces 33° above the horizontal, the intensity of the difference image calculated by the same method as in FIG. It is a diagram in which a double circle indicating the neck position and a black circle indicating the chin position are superimposed on each other by adjusting the scale according to the strength. 4 is a timing chart showing lighting timings of infrared LEDs and related signals; It is a figure explaining a motion of a user's face of the up-down direction. FIG. 10 is a diagram showing a target field of view in a super-wide-angle image captured by the imaging unit of the camera body when the user faces the front; 11B is a diagram showing an image of a target field of view cut out from the ultra-wide-angle image of FIG. 11A. FIG. 4 is a diagram showing a target field of view in a super-wide-angle image when a user observes subject A; FIG. FIG. 11C is a diagram showing an image obtained by correcting distortion and shaking with respect to the image of the target field of view in FIG. 11C cut out from the ultra-wide-angle image. FIG. 11B is a diagram showing a target field of view in a super-wide-angle image when the user observes the subject A with a field angle set value smaller than that in FIG. 11C. FIG. 11C is a diagram showing an image obtained by correcting distortion and shaking with respect to the image of the target field of view in FIG. 11E cut out from the ultra-wide-angle image. FIG. 4 is a diagram showing an example of a target field of view in a super-wide-angle image; 12B is a diagram showing an example of a target field of view in a super-wide-angle image, which has the same field angle setting value as the target field of view in FIG. 12A but has a different observation direction. FIG. 12B is a diagram showing another example of a target field of view in a super-wide-angle image having the same field angle setting value as the target field of view of FIG. 12A but with a different observation direction. FIG. FIG. 12B is a diagram showing an example of a target field of view in a super-wide-angle image, which has the same observation direction as the target field of view of FIG. 12C but has a smaller field angle setting value. FIG. 12B is a diagram showing an example in which a preliminary region for image stabilization corresponding to a predetermined image stabilization level is provided around the target field of view shown in FIG. 12A; FIG. 12C is a diagram showing an example in which an anti-vibration preliminary area corresponding to the same anti-vibration level as that of the anti-vibration preliminary area of FIG. 12E is provided around the target visual field shown in FIG. 12B. FIG. 12C is a diagram showing an example in which an anti-vibration preliminary area corresponding to the same anti-vibration level as that of the anti-vibration preliminary area of FIG. 12E is provided around the target field of view shown in FIG. 12D. FIG. 4 is a diagram showing a menu screen for various settings of a moving image mode displayed on the display unit of the display device before the camera main body captures an image; 7B is a flowchart of a subroutine of primary recording processing in step S600 of FIG. 7A; FIG. 4 is a diagram showing the data structure of a video file generated by primary recording processing; FIG. 7B is a flowchart of a subroutine of transfer processing to the display device in step S700 of FIG. 7A; FIG. 7B is a flowchart of a subroutine for optical correction processing in step S800 of FIG. 7A; FIG. 18 is a diagram for explaining the process of performing distortion aberration correction in step S803 of FIG. 17; FIG. FIG. 7B is a flowchart of a subroutine of image stabilizing processing in step S900 of FIG. 7A; FIG. FIG. 10 is a diagram showing details of a calibrator used for calibration processing according to the second embodiment; 10 is a flowchart of calibration processing according to Example 2, which is executed in a camera body and a calibrator; FIG. 22 is a diagram showing a screen displayed on the display unit of the calibrator in step S3103 of FIG. 21 during the calibration operation in the front direction of the user; FIG. 22B is a diagram showing a user holding the calibrator forward according to the instructions shown in the instruction display in FIG. 22A. FIG. 22B is a schematic diagram showing the entire super-wide-angle image captured by the imaging lens in the state of FIG. 22B; FIG. 22C is a schematic diagram showing an image in which the super-wide-angle image shown in FIG. 22C is corrected for aberration; FIG. 22 is a schematic diagram showing a face direction image acquired by the face direction detection unit in step S3108 of FIG. 21 during the calibration operation for the front direction of the user; FIG. 22 is a schematic diagram showing an in-camera image displayed in step S3107 of FIG. 21; FIG. 22 is a diagram showing a screen displayed on the display unit of the calibrator in step S3103 of FIG. 21 during the calibration operation in the upper right direction of the user; FIG. 23B is a diagram showing a state in which the user holds the calibrator up and right according to the instruction shown in the instruction display in FIG. 23A. 23B is a schematic diagram showing the entire super-wide-angle image captured by the imaging lens in the state of FIG. 23B; FIG. FIG. 23D is a schematic diagram showing an image in which the ultra-wide-angle image shown in FIG. 23C is corrected for aberration; FIG. 22 is a schematic diagram showing a face direction image acquired by the face direction detection unit in step S3108 of FIG. 21 during the calibration operation for the upper right direction of the user; FIG. 11 is a diagram for explaining delayed image clipping in Example 3; FIG. 12 is a diagram showing a trajectory of facial movement retained in Example 3; 10 is a flowchart of motion sickness prevention processing according to Example 3. FIG. FIG. 16 is a graph for explaining a clipping range correction process according to Example 4; FIG. (a) is a flow chart showing recording direction/range determination processing according to the fourth embodiment, and (b) is a flow chart showing extraction range correction processing in step S400 of (a). FIG. 10 is a schematic diagram for explaining the relationship between the user's field of view and the target field of view when an object at a short distance is an observation target in Example 1; FIG. 11 is an external view of a camera body including an imaging device according to Example 5; FIG. 12 is a block diagram showing the hardware configuration of a camera body according to Example 5; FIG. 11 is a schematic diagram for explaining the relationship between a user, a calibrator, and a target visual field at the time of calibration including parallax correction mode processing in Example 5; 10 is a flowchart of parallax correction mode processing that is part of the preparation processing in step S100 of FIG. 7A according to Example 5. FIG. 10 is a flowchart of a subroutine of the recording direction/range determination process of S300 described with reference to FIG. 7A according to the fifth embodiment; FIG. 33B is a schematic diagram showing the relationship between the defocus map created in step S5302 of FIG. 33B and the recording direction; 14 is a flowchart of observation direction determination processing according to Example 6. FIG. FIG. 11 is a diagram showing the relationship between a user's viewing direction detection state for each frame and a captured image according to Example 6; FIG. 12 is a diagram showing the relationship between the observation direction detection state of the user for each frame and the captured image in the subject lost mode according to Example 6; FIG. 16 is a diagram for explaining the relationship between the observation direction and the face area that can be used for detection of the face direction according to the seventh embodiment; FIG. 36 is a flowchart of viewing direction determination processing when obtaining a face direction according to Embodiment 7, which is performed instead of the processing in step S6004 of FIG. 35; FIG. 21 is a diagram showing the relationship between face direction and face direction reliability in Example 7; FIG. 21 is a conceptual diagram of observation direction determination processing when obtaining a face direction in Embodiment 7; It is an enlarged view showing a mode that the imaging / detection part was seen from the side. FIG. 4 is a side view showing how a user wears the camera body; FIG. 11 is an enlarged view showing a side view of the photographing/detecting unit when the connection unit is hidden; FIG. 11 is a side view showing a state in which a user wears the camera body when the connection portion is hidden; FIG. 4 is a view showing a connection surface, which is a cut surface of a band portion and an electric cable integrally formed therewith; FIG. 21 is a block diagram showing the hardware configuration of a display device connected to a camera body including an imaging device according to Example 9; FIG. 20 is a functional block diagram of a camera body according to Example 9; FIG. 20 is a functional block diagram of a camera body and a display device according to Example 10; FIG. 21 is a flow chart showing an overview of image recording processing according to the tenth embodiment, which is executed in a camera body and a display device; FIG. FIG. 20 is a functional block diagram of a camera body and a display device according to Example 11; FIG. 21 is a flow chart showing an outline of image recording processing according to an eleventh embodiment, which is executed in a camera body and a display device; FIG. FIG. 21 is an external view of a camera body in Example 12; FIG. 21 is a perspective view showing the details of an imaging/detecting section which is a part of a camera body in Embodiment 12; FIG. 51C is a perspective view showing a state in which the photographing unit of the photographing/detecting unit of FIG. 51B is rotated 30 degrees to the left; FIG. 51C is a perspective view showing a state in which the imaging unit of FIG. 51C faces downward 30 degrees; FIG. 20 is a functional block diagram of a camera body according to Example 12; FIG. 22 is a block diagram showing the hardware configuration of a camera body according to Example 12; FIG. 20 is a flow chart showing an outline of image recording processing according to the twelfth embodiment, which is executed in a camera body and a display device; FIG. 55 is a flowchart of a subroutine of imaging unit drive processing in step S12300 of FIG. 54 according to Example 12. FIG. 55 is a flowchart of a subroutine of development processing in step S12500 of FIG. 54 according to Example 12. FIG. FIG. 22 is a block diagram showing the hardware configuration of a camera body according to Example 13; FIG. 21 is a schematic diagram showing an example of a learning image used in Example 13; 29 is a flow chart showing processing for detecting face direction using machine learning according to the thirteenth embodiment. FIG. 20 is a block diagram showing the hardware configuration of a camera body according to Example 14; (a) is a schematic diagram showing a distance image generated when the ToF device of the camera body according to Example 14 is installed at the collarbone position and the ToF device is used to measure and photograph the upward direction; (a) is a schematic diagram showing an image obtained by subjecting the distance image to threshold processing to extract a face portion, (c) is a schematic diagram showing an image obtained by dividing the image (b) into areas according to distance information, and (d). FIG. 11 is a schematic diagram showing an image showing a neck position and a chin position extracted from the image of (c); FIG. 22 is a flow chart showing face direction calculation processing in the fourteenth embodiment. FIG. 1 is a diagram showing a configuration example of a camera fixed to the head using a conventional head fixing accessory; FIG. 1 is a diagram showing a configuration example of a conventional omnidirectional imaging camera; FIG. FIG. 49 is a diagram showing an example of conversion work of a video imaged by the omnidirectional camera of FIG. 48;

Preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

First, we consider some ways to solve the dual problem of capturing important moments and keeping the photographer focused on the experience. One method is to fix an action camera to the head using a head fixation accessory and take an image in the observing direction, so that the photographer can take an image without taking his or her hand to perform the imaging operation. Also, by shooting a wide range with a omnidirectional camera, you can concentrate on the experience during the experience, and after the experience is over, cut out the necessary video parts from the captured omnidirectional video and edit them. There is also a way to leave a video of what you did.

The former method requires the troublesome act of wearing a head fixing accessory 902 to which the main body of the action camera 901 is fixed, as shown in FIG. 63(a). Also, as shown in FIG. 63(b), when the photographer attaches the action camera 901 to the head with the head-fixing accessory 902, the appearance is not good, and problems such as the hairstyle of the photographer being disturbed occur. Furthermore, the photographer may be concerned about the weight of the head fixing accessory 902 and the action camera 901 worn on the head, or may be concerned about the unattractive appearance of the third person. was Therefore, in the state shown in FIG. 63(b), the photographer cannot concentrate on the experience, or the photographer feels resistance to the state shown in FIG. 63(b). I had a problem with it being tough.

On the other hand, the latter method requires a series of operations such as image conversion and clipping position designation. For example, as shown in FIG. 64, an omnidirectional imaging camera 903 having a lens 904 and an imaging button 905 is known. The lens 904 is one of a pair of fish-eye lenses for half-sphere shooting arranged on both sides of the housing of the omnidirectional camera 903, and the omnidirectional camera 903 uses this pair of fish-eye lenses to perform omnidirectional imaging. Take ball shots. Then, an omnidirectional image is obtained by synthesizing the captured images using the pair of fisheye lenses.

65A and 65B are diagrams showing an example of the conversion work of the video imaged by the omnidirectional imaging camera 903. FIG.

FIG. 65(a) is an example of an omnidirectional image obtained by the omnidirectional imaging camera 903, and includes a photographer 906, a child 907, and a tree 908 as subjects. Since this image is an omnidirectional image obtained by synthesizing projection images of a pair of fisheye lenses, the image of the photographer 906 is greatly distorted. A child 907, which is the subject that the photographer 906 was trying to capture, has a body part in the periphery of the imaging range of the lens 904, so the body part is greatly distorted left and right and stretched. On the other hand, the tree 908 is an object positioned in front of the lens 904, so it is imaged without significant distortion.

From the image in FIG. 65(a), in order to create an image of the field of view that people usually see, it is necessary to cut out a portion of the image, convert it into a plane, and display it.

FIG. 65(b) is an image obtained by cutting out an image located in front of the lens 904 from the image of FIG. 65(a). In the image of FIG. 65(b), a tree 908 is shown in the center in a field of view that people usually see. However, since the child 907 that the photographer 906 was trying to capture is not included in the image in FIG. 65(b), the clipping position must be changed. Specifically, it is necessary to set the cutting position to the left and 30° downward from the tree 908 as viewed in FIG. 65(a). FIG. 65(c) shows an image displayed after performing the clipping operation and then planarly converted. As described above, in order to obtain the image shown in FIG. 65(c), which the photographer intended to capture, from the image shown in FIG. For this reason, the photographer can concentrate on the experience during the experience (during imaging), but there is a problem that the amount of work after that becomes enormous. Therefore, a configuration like the first embodiment was considered to solve these problems.

(Example 1)
1A to 1D are diagrams for explaining a camera system comprising a camera body 1 including an imaging/detecting unit 10 as a wearable imaging device according to the present embodiment and a display device 800 configured separately from this. is. In this embodiment, the camera body 1 and the display device 800 are shown as separate bodies, but they may be constructed integrally. A user wearing the camera body 1 around the neck is hereinafter referred to as a user.

FIG. 1A is an external view of the camera body 1. FIG.

In FIG. 1A, the camera body 1 includes a photographing/detecting section 10, a battery section 90 (power supply means), a right connecting section 80R, and a left connecting section 80L. The right connecting portion 80R connects the imaging/detecting portion 10 and the battery portion 90 on the right side of the user's body (on the left side as viewed in FIG. 1A). The left connection portion 80L connects the imaging/detection portion 10 and the battery portion 90 on the left side of the user's body (on the right side as viewed in FIG. 1A).

The photographing/detecting unit 10 includes a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, an LED 17, and microphones 19L and 19R.

The face direction detection window 13 detects infrared rays emitted from an infrared LED 22 ( FIG. 5 : infrared irradiation means) for detecting the positions of each part of the user's face, which is built in the photographing/detecting section 10, and its reflection. Allows light to pass through.

The start switch 14 is a switch for starting imaging.

A stop switch 15 is a switch for stopping imaging.

The imaging lens 16 guides light rays to be imaged into the imaging/detecting unit 10 and forms an optical image on the solid-state imaging device 42 (FIG. 5).

The LED 17 is an LED that indicates that an image is being captured or indicates a warning.

The microphones 19R and 19L are microphones that take in surrounding sounds. The microphone 19L takes in sounds from the left side of the user's surroundings (the right side in FIG. 1A), and the microphone 19R takes in the sounds from the right side of the user's surroundings (the right side in FIG. 1A). to the left).

FIG. 1B is a diagram showing how the camera body 1 is hung by the user.

When the battery unit 90 is attached to the user's back side and the photographing/detecting unit 10 is attached to the front side of the user's body, both ends are connected near the left and right ends of the photographing/detecting unit 10 Left and right connection The parts 80L and 80R are biased and supported in the chest direction. As a result, the photographing/detecting unit 10 is positioned in front of the user's clavicle. At this time, the face direction detection window 13 is located under the chin of the user. Within the face orientation detection window 13 is an infrared condensing lens 26 which is illustrated later in FIG. 2E. The optical axis (detection optical axis) of the infrared condensing lens 26 faces the user's face and faces in a different direction from the optical axis (imaging optical axis) of the imaging lens 16 . A face direction detection unit 20 (face direction detection means, see FIG. 5) including an infrared condensing lens 26 detects the viewing direction of the user from the position of each part of the face. As a result, an imaging unit 40 (imaging means), which will be described later, can capture an image in the observation direction.

How to adjust the set position according to individual differences in body shape and clothes will be described later.

In addition, by arranging the photographing/detecting unit 10 on the front of the body and the battery unit 90 on the back, the weight is distributed, the fatigue of the user is reduced, and the displacement due to centrifugal force or the like when the user moves. has the effect of suppressing

In this embodiment, an example is shown in which the imaging/detecting unit 10 is attached so as to be positioned in front of the collarbone of the user, but the present invention is not limited to this. That is, if the camera body 1 can detect the user's observation direction by the face direction detection unit 20 and can capture an image in the observation direction by the imaging unit 40, the camera body 1 can be used on the user's body other than the head. may be attached to any of the

FIG. 1C is a view of the battery section 90 viewed from the rear of FIG. 1A.

In FIG. 1C, the battery section 90 includes a charging cable insertion port 91, adjustment buttons 92L and 92R, and a notch 93 for protecting the spine.

The charging cable insertion port 91 is an insertion port for a charging cable (not shown). Through this charging cable, internal batteries 94L and 94R (see FIG. 3A) are charged from an external power source, and power is supplied to the imaging/detecting unit 10. or

The adjustment buttons 92L, 92R are buttons for adjusting the length of the band portions 82L, 82R of the left and right connection portions 80L, 80R. The adjustment button 92L is a button for adjusting the left band portion 82L, and the adjustment button 92R is a button for adjusting the right band portion 82R. In this embodiment, the lengths of the band portions 82L and 82R are adjusted independently by the adjustment buttons 92L and 92R, but the lengths of the band portions 82L and 82R are adjusted simultaneously by one button. good too.

The spine avoidance notch 93 is a notch portion that avoids the spine portion so that the battery portion 90 does not come into contact with the spine portion of the user. By avoiding the convex part of the spine of the human body, it reduces the discomfort of wearing it, and at the same time prevents the body from moving left and right during use.

FIG. 1D is an external view of a display device 800 as a mobile device according to the first embodiment, which is configured separately from the camera body 1. FIG.

1D, the display device 800 includes a button A802, a display unit 803, a button B804, an in-camera 805, a face sensor 806, an angular velocity sensor 807, and an acceleration sensor 808. FIG. Also, although not shown in FIG. 1D, a wireless LAN capable of high-speed connection with the camera body 1 is provided.

A button A 802 is a button having the function of the power button of the display device 800 . The display device 800 accepts a power ON/OFF operation by long-pressing the button A802, and accepts other processing timing instructions by short-pressing the button A802.

A display unit 803 can check an image captured by the camera body 1 and display a menu screen necessary for setting. In this embodiment, a transparent touch sensor is also provided on the upper surface of the display unit 803, and receives an operation by touching a screen being displayed (for example, a menu screen).

The button B804 is a button that functions as a calibration button 854 used for calibration processing, which will be described later.

The in-camera 805 is a camera capable of capturing an image of a person observing the display device 800 .

A face sensor 806 detects the face shape and viewing direction of a person viewing the display device 800 . Although the specific structure of the face sensor 806 is not particularly limited, it can be implemented with various sensors such as a structured light sensor, a ToF sensor, and a millimeter wave radar.

Since the angular velocity sensor 807 is inside the display device 800, it is indicated by a dotted line as a perspective view. Since the display device 800 of the present embodiment also has a function of a calibrator, which will be described later, it is equipped with gyro sensors in the three-dimensional X, Y, and Z directions.

An acceleration sensor 808 detects the orientation of the display device 800 .

A general smart phone is used for the display device 800 according to the present embodiment, and the camera system according to the present invention can be implemented by making the firmware in the smart phone compatible with the firmware on the camera body 1 side. there is However, it is also possible to implement the camera system according to the present invention by adapting the firmware of the camera body 1 side to the application and OS of the smartphone as the display device 800 .

2A to 2F are diagrams illustrating the imaging/detection unit 10 in detail. In the subsequent figures, the same numbers are assigned to the parts that have already been explained, meaning the same functions, and the explanation in this specification is omitted.

FIG. 2A is a front view of the photographing/detecting unit 10. FIG.

The right connecting portion 80R has an angle holding portion 81R and a band portion 82R made of a hard material for holding an angle with respect to the photographing/detecting portion 10, and the left connecting portion 80L has an angle holding portion 81L and a band portion 82L.

FIG. 2B is a diagram showing the shape of the band portions 82L, 82R of the left and right connection portions 80L, 80R. In this figure, the angle holding portions 81L and 81R are seen through in order to show the shapes of the band portions 82L and 82R.

The band portion 82L includes a left connection surface 83L arranged on the left side of the user's body (the right side in FIG. 2B) when the camera body 1 is worn, and an electric cable 84. As shown in FIG. The band portion 82R includes a right connecting surface 83R that is arranged on the right side of the user's body (the left side in FIG. 2B) when the camera body 1 is worn.

The left connecting surface 83L is connected to the angle holding portion 81L and has a non-circular cross-sectional shape, here an elliptical shape. The right connecting surface 83R also has a similar elliptical shape. The right connection surface 83R and the left connection surface 83L are shaped like the Japanese katakana character "C". That is, the distance between the symmetrically positioned portions of the right connection surface 83R and the left connection surface 83L becomes closer from the bottom to the top in FIG. 2B. As a result, when the user hangs the camera body 1, the long axis direction of the left and right connecting surfaces 83L and 83R is aligned with the user's body. There is an effect that the photographing/detecting unit 10 does not move in the left, right, front, and rear directions while being comfortable when in contact with it.

An electric cable 84 (power supply means) is wired inside the band portion 82L and electrically connects the battery portion 90 and the photographing/detecting portion 10 . The electric cable 84 connects the power source of the battery section 90 to the photographing/detecting section 10 and transmits/receives electric signals to/from the outside.

FIG. 2C is a diagram of the photographing/detecting unit 10 as seen from the back side. Since FIG. 2C is a view from the side in contact with the user's body, that is, the opposite side of FIG. 2A, the positional relationship between the right connecting portion 80R and the left connecting portion 80L is reversed from that in FIG. 2A.

The photographing/detecting unit 10 has a power switch 11, an imaging mode switch 12, and chest connection pads 18a and 18b on its back side.

The power switch 11 is a power switch for switching ON/OFF of the power of the camera body 1 . The power switch 11 of this embodiment is a switch in the form of a slide lever, but is not limited to this. For example, the power switch 11 may be a push-type switch, or may be a switch integrated with a slide cover (not shown) of the imaging lens 16 .

The imaging mode switch 12 (changer) is a switch for changing the imaging mode, and can change the mode related to imaging. In this embodiment, the imaging mode switch 12 can switch to a preset mode set using the display device 800, which will be described later, in addition to the still image mode and moving image mode. In this embodiment, the imaging mode switch 12 is a switch in the form of a slide lever that can select one of "Photo", "Normal", and "Pre" shown in FIG. 2C by sliding the lever. The imaging mode shifts to still image mode by sliding to "Photo", shifts to moving image mode by sliding to "Normal", and shifts to preset mode by sliding to "Pre". Note that the imaging mode switch 12 is not limited to the mode of this embodiment as long as it is a switch that can change the imaging mode. For example, the imaging mode switch 12 may be composed of three buttons "Photo", "Normal", and "Pre".

The chest connection pads 18a and 18b (fixing means) are parts that come into contact with the user's body when the imaging/detection unit 10 is urged against the user's body. As shown in FIG. 2A, the photographing/detecting unit 10 is shaped so that its horizontal (left and right) length is longer than its vertical (upper and lower) length when worn, and the chest connection pads 18a and 18b are attached to the photographing/detecting unit. 10 are arranged near the left and right ends. By arranging in this way, it becomes possible to suppress left-right rotational blurring during imaging by the camera body 1 . Moreover, the presence of the chest connection pads 18a and 18b can prevent the power switch 11 and the imaging mode switch 12 from coming into contact with the body. Furthermore, the chest connection pads 18a and 18b serve to prevent the heat from being transmitted to the user's body even if the temperature of the imaging/detecting unit 10 rises due to long-time imaging, and to adjust the angle of the imaging/detecting unit 10. also plays the role of

FIG. 2D is a top view of the photographing/detecting unit 10. FIG.

As shown in FIG. 2D , a face direction detection window 13 is provided in the center of the upper surface of the photographing/detecting section 10 , and chest connection pads 18 a and 18 b protrude from the photographing/detecting section 10 .

FIG. 2E is a diagram showing the configuration of the face direction detection section 20 which is arranged inside the photographing/detection section 10 and below the face direction detection window 13 .

The face direction detection unit 20 includes an infrared LED 22 and an infrared condensing lens 26 . The face direction detection unit 20 further includes an infrared LED lighting circuit 21 and an infrared detection processing device 27 shown in FIG. 5, which will be described later.

The infrared LED 22 projects infrared rays 23 (FIG. 5) toward the user.

The infrared condensing lens 26 is a lens that forms an image of the reflected light beam 25 (FIG. 5) reflected from the user when the infrared ray 23 is projected from the infrared LED 22 on a sensor (not shown) of the infrared detection processing device 27. .

FIG. 2F is a view of the state in which the user hangs the camera body 1 as viewed from the left side of the user.

The angle adjustment button 85L is provided in the angle holding section 81L and used when adjusting the angle of the photographing/detecting section 10. FIG. Although not shown in this figure, an angle adjustment button is also set inside the angle holding portion 81R on the opposite side at a position symmetrical to the angle adjustment button 85L.

The angle adjustment buttons are also visible in FIGS. 2A, 2C, and 2D, but are omitted for simplicity of explanation.

The user can change the angle between the photographing/detecting section 10 and the angle holding section 81L by moving the angle holding section 81L up and down toward FIG. 2F while pressing the angle adjustment button 85L. The same is true for the right side. Moreover, the chest connection pads 18a and 18b can be changed in projection angle. The photographing/detecting unit 10 horizontally adjusts the optical axis of the imaging lens 16 by the action of these two types of angle changing members (the angle adjustment button and the chest connection pad) regardless of individual differences in the shape of the chest position of the user. It is possible.

FIG. 3 is a diagram illustrating the details of the battery section 90. As shown in FIG.

FIG. 3A is a partially see-through view of the battery section 90 from the back.

As shown in FIG. 3(a), the battery section 90 has a left battery 94L and a right battery 94R symmetrically mounted therein in order to balance its weight. By symmetrically arranging the left and right batteries 94L and 94R with respect to the central portion of the battery section 90 in this manner, the left and right weight balances are balanced and the positional deviation of the camera body 1 is prevented. Note that the battery section 90 may have a configuration in which only one battery is mounted.

FIG. 3(b) is a top view of the battery section 90. As shown in FIG. Also in this figure, the batteries 94L and 94R are shown transparently.

As shown in FIG. 3(b), by symmetrically arranging the batteries 94L and 94R on both sides of the notch 93 for avoiding the spine, the relatively heavy battery section 90 can be worn by the user without burden. is possible.

FIG. 3C is a diagram of the battery section 90 viewed from the back side. FIG. 3(c) is a view seen from the side in contact with the user's body, that is, the opposite side of FIG. 3(a).

As shown in FIG. 3(c), the spine-relief notch 93 is centrally provided along the user's spine.

FIG. 4 is a functional block diagram of the camera body 1. As shown in FIG. Since the details will be described later, the general flow of processing executed by the camera body 1 will be described here with reference to FIG.

4, the camera body 1 includes a face direction detection unit 20, a recording direction/angle of view determination unit 30, a photographing unit 40, an image clipping/development processing unit 50, a primary recording unit 60, a transmission unit 70, and other control unit 111. Prepare. These functional blocks are executed under the control of an overall control CPU 101 (FIG. 5) that performs overall control of the camera body 1. FIG.

The face direction detection unit 20 (observation direction detection means) is a functional block executed by the infrared LED 22, the infrared detection processing device 27, etc., detects the face direction, infers the observation direction, and records it. It is passed to the direction/angle of view determination unit 30 .

The recording direction/angle of view determining unit 30 (recording direction determining means) performs various calculations based on the observation direction inferred by the face direction detecting unit 20, and obtains information on the position and range when extracting the image from the photographing unit 40. is determined, and this information is transferred to the image clipping/development processing unit 50 .

The photographing unit 40 converts the light beam from the object into a wide-angle image, and transfers the image to the image clipping/development processing unit 50 .

The image clipping/developing processing unit 50 (developing means) uses the information from the recording direction/angle of view determining unit 30 to extract and develop only the image in the direction in which the user is looking from the image from the photographing unit 40. , to the primary recording unit 60 .

The primary recording unit 60 is a functional block configured by the primary memory 103 (FIG. 5) and the like, records video information, and transfers it to the transmitting unit 70 at the necessary timing.

The transmission unit 70 (video output means) is wirelessly connected to the display device 800 (FIG. 1D), the calibrator 850, and the simple display device 900, which are predetermined communication partners, and communicates with them.

The display device 800 can be connected to the transmitter 70 via a wireless LAN (hereinafter referred to as "high-speed wireless") that allows high-speed connection. Here, in this embodiment, wireless communication corresponding to the IEEE802.11ax (WiFi 6) standard is used for high-speed wireless communication, but wireless communication corresponding to other standards such as the WiFi 4 standard and the WiFi 5 standard is used. good too. Further, the display device 800 may be a device developed exclusively for the camera body 1, or may be a general smart phone, a tablet terminal, or the like.

The transmission unit 70 and the display device 800 may be connected using low-power radio, or may be connected by both high-speed radio and low-power radio, or may be connected by switching. In this embodiment, a large amount of data, such as a video file of a moving image, which will be described later, is transmitted by high-speed wireless transmission, and light data or data that may take a long time to transmit is transmitted by low-power wireless transmission. In this embodiment, Bluetooth is used for low-power wireless communication, but other short-range wireless communication such as NFC (Near Field Communication) may be used.

The calibrator 850 is a device for performing initial settings of the camera body 1 and individual settings, and can be connected to the transmission unit 70 by high-speed wireless communication, like the display device 800 . Details of the calibrator 850 will be described later. Also, the display device 800 may also function as the calibrator 850 .

The simple display device 900 is a display device that can be connected to the transmitter 70 only by low-power radio, for example.

The simple display device 900 cannot transmit moving images with the transmission unit 70 due to time restrictions, but can perform timing transmission of imaging start/stop and image confirmation such as composition confirmation. As with the display device 800, the simple display device 900 may be a device developed exclusively for the camera body 1, or may be a smart watch or the like.

FIG. 5 is a block diagram showing the hardware configuration of the camera body 1. As shown in FIG. Further, the configurations and functions described with reference to FIGS. 1A to 1C and the like are denoted by the same reference numerals, and detailed description thereof will be omitted.

5, the camera body 1 includes an overall control CPU 101, a power switch 11, an imaging mode switch 12, a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, and an LED 17.

The camera body 1 also includes an infrared LED lighting circuit 21, an infrared LED 22, an infrared condensing lens 26, and an infrared detection processing device 27, which constitute the face direction detection section 20 (FIG. 4).

The camera body 1 also includes an imaging unit 40 (FIG. 4) comprising an imaging driver 41, a solid-state imaging device 42, and an imaging signal processing circuit 43, and a transmitting unit 70 (FIG. 4) comprising a low-power wireless unit 71 and a high-speed wireless unit 72. 4).

Although only one imaging unit 40 is provided in the present embodiment, the camera body 1 is provided with two or more imaging units 40 to capture a 3D image or an image that can be acquired by one imaging unit 40. A wide-angle image may be captured, or images may be captured in a plurality of directions.

The camera body 1 also includes various memories such as a large-capacity nonvolatile memory 51 , an internal nonvolatile memory 102 , and a primary memory 103 .

Further, the camera body 1 includes an audio processing section 104 , a speaker 105 , a vibrating body 106 , an angular velocity sensor 107 , an acceleration sensor 108 and various switches 110 .

The overall control CPU 101 is connected to the power switch 11 described above with reference to FIG. 2C, and controls the camera body 1 . The recording direction/angle of view determination unit 30, image clipping/development processing unit 50, and other control unit 111 shown in FIG. 4 are implemented by the overall control CPU 101 itself.

The infrared LED lighting circuit 21 controls lighting and extinguishing of the infrared LED 22 described above with reference to FIG. 2E, and controls projection of infrared rays 23 from the infrared LED 22 toward the user.

The face direction detection window 13 is composed of a visible light cut filter, which hardly transmits visible light, but sufficiently transmits infrared ray 23 which is light in the infrared region and its reflected light 25 .

The infrared condensing lens 26 is a lens that condenses the reflected light beam 25 .

The infrared detection processing device 27 (infrared detection means) has a sensor that detects the reflected light beam 25 condensed by the infrared condensing lens 26 . This sensor converts the image formed by the collected reflected light beam 25 into sensor data and transfers it to the overall control CPU 101 .

When the user hangs the camera body 1 as shown in FIG. 1B, the face direction detection window 13 is located under the user's chin. Therefore, the infrared rays 23 projected from the infrared LED 22 pass through the face direction detection window 13 and are irradiated onto the infrared irradiation surface 24 near the chin of the user as shown in FIG. A reflected light ray 25 reflected by the infrared irradiation surface 24 passes through the face direction detection window 13 and is condensed by an infrared condensing lens 26 to a sensor in an infrared detection processing device 27 .

The various switches 110 are not shown in FIGS. 1A to 1C and the like, and although the details are omitted, they are switches for executing functions unrelated to this embodiment.

The imaging driver 41 includes a timing generator and the like, generates and outputs various timing signals to each unit related to imaging, and drives the solid-state imaging device 42 .

The solid-state imaging device 42 outputs a signal obtained by photoelectrically converting the subject image projected from the imaging lens 16 described with reference to FIG. 1A to the imaging signal processing circuit 43 .

The imaging signal processing circuit 43 outputs imaging data generated by performing processing such as clamping and processing such as A/D conversion on the signal from the solid-state imaging device 42 to the overall control CPU 101 .

The built-in non-volatile memory 102 is a flash memory or the like, and stores a startup program for the overall control CPU 101 and setting values for various program modes. In this embodiment, since the observation field of view (angle of view) and the effect level of image stabilizing control can be set, such setting values are also recorded.

The primary memory 103 is composed of a RAM or the like, and temporarily stores video data being processed and temporarily stores the calculation results of the overall control CPU 101 .

A large-capacity nonvolatile memory 51 stores image data. In this embodiment, the large-capacity nonvolatile memory 51 is a non-removable semiconductor memory. However, the large-capacity nonvolatile memory 51 may be composed of a removable recording medium such as an SD card, or may be used together with the built-in nonvolatile memory 102 .

The low-power radio unit 71 exchanges data with the display device 800, the calibrator 850, and the simple display device 900 by low-power radio.

The high-speed wireless unit 72 exchanges data with the display device 800 and the calibrator 850 by high-speed wireless.

The audio processing unit 104 processes external sounds (analog signals) picked up by the microphones 19L and 19R to generate audio signals.

The LED 17, the speaker 105, and the vibrator 106 emit light, emit sound, or vibrate to notify or warn the user of the state of the camera body 1. FIG.

The angular velocity sensor 107 is a sensor using a gyro or the like, and detects movement of the camera body 1 itself as gyro data.

The acceleration sensor 108 detects the orientation of the imaging/detecting unit 10 .

FIG. 6 is a block diagram showing the hardware configuration of the display device 800. As shown in FIG. For simplification of explanation, the parts explained with reference to FIG. 1D are denoted by the same reference numerals and the explanation thereof is omitted.

6, the display device 800 includes a display device control unit 801, a button A802, a display unit 803, a button B804, a face sensor 806, an angular velocity sensor 807, an acceleration sensor 808, an imaging signal processing circuit 809, and various switches 811.

The display device 800 also includes a built-in non-volatile memory 812, a primary memory 813, a large-capacity non-volatile memory 814, a speaker 815, a vibrating body 816, an LED 817, an audio processing section 820, a low-power radio unit 871, and a high-speed radio unit 872. Prepare. Each element described above is connected to the display device control unit 801 .

A display device control unit 801 is configured by a CPU and controls the display device 800 .

The imaging signal processing circuit 809 has functions equivalent to those of the imaging driver 41, the solid-state imaging device 42, and the imaging signal processing circuit 43 inside the camera body 1, and constitutes the in-camera 805 of FIG. 1D together with the in-camera lens 805a. . Data output from the imaging signal processing circuit 809 is processed within the display device control unit 801 . Details of processing of this data will be described later.

Various switches 811 are switches for executing functions unrelated to this embodiment.

An angular velocity sensor 807 is a sensor using a gyro or the like, and detects movement of the display device 800 itself.

An acceleration sensor 808 detects the orientation of the display device 800 itself.

A built-in non-volatile memory 812 uses a flash memory or the like, and stores a startup program for the display device control unit 801 and setting values for various program modes.

A primary memory 813 is configured by a RAM or the like, and temporarily stores video data being processed and temporarily stores the calculation results of the imaging signal processing circuit 809 . In this embodiment, during video recording, the gyro data detected by the angular velocity sensor 107 at the imaging time of each frame is associated with each frame and held in the primary memory 813 .

A large-capacity non-volatile memory 814 stores image data of the display device 800 . In this embodiment, the large-capacity non-volatile memory 814 is composed of removable memory such as an SD card. A non-detachable memory such as the large-capacity non-volatile memory 51 in the camera body 1 may be used.

The speaker 815, vibrator 816, and LED 817 emit sound, vibrate, or emit light to notify or warn the user of the state of the display device 800. FIG.

The audio processing unit 820 processes external sounds (analog signals) picked up by the left microphone 819L and the right microphone 819R to generate audio signals.

The low-power radio unit 871 exchanges data with the camera body 1 by low-power radio.

The high-speed radio unit 872 exchanges data with the camera body 1 by high-speed radio.

The face sensor 806 (face detection means) includes an infrared LED lighting circuit 821 , an infrared LED 822 , an infrared condenser lens 826 and an infrared detection processing device 827 .

The infrared LED lighting circuit 821 is a circuit having the same function as the infrared LED lighting circuit 21 in FIG. to control the light emission of the

The infrared condensing lens 826 is a lens that condenses the reflected light beam 825 of the infrared rays 823 .

Infrared detection processing device 827 has a sensor that detects the reflected light beams collected by infrared collecting lens 826 . This sensor converts the collected reflected light beam 825 into sensor data and passes it to the display device controller 801 .

When the face sensor 806 shown in FIG. 1D is directed toward the user, as shown in FIG. 6, infrared rays 823 projected from the infrared LED 822 are irradiated onto the infrared irradiation surface 824, which is the entire face of the user. A reflected light ray 825 reflected by the infrared irradiation surface 824 is condensed by an infrared condensing lens 826 to a sensor in an infrared detection processing device 827 .

The other function unit 830 executes smartphone functions such as telephone functions that are not related to this embodiment.

How to use the camera body 1 and the display device 800 will be described below.

FIG. 7A is a flow chart showing an outline of image recording processing according to the present embodiment, which is executed in the camera body 1 and the display device 800. FIG.

As an aid to the explanation, in FIG. 7A, which device shown in FIG. 4 is performing the step is described on the right side of each step. That is, steps S100 to S700 in FIG. 7A are executed by the camera body 1, and steps S800 to S1000 in FIG. 7A are executed by the display device 800. FIG.

When the power switch 11 is turned on and the camera body 1 is powered on, the general control CPU 101 is activated and reads out the activation program from the built-in non-volatile memory 102 . Thereafter, in step S100, the overall control CPU 101 executes preparatory operation processing for setting the camera body 1 before imaging. Details of the preparatory operation process will be described later with reference to FIG. 7B.

In step S200, the face direction detection unit 20 detects the face direction, and performs face direction detection processing for analogizing the viewing direction. The details of the face direction detection process will be described later with reference to FIG. 7C. This process is executed at a predetermined frame rate.

In step S300, the recording direction/angle of view determination unit 30 executes recording direction/range determination processing. Details of the recording direction/range determination process will be described later with reference to FIG. 7D.

In step S400, the imaging unit 40 performs imaging and generates imaging data.

In step S500, the image clipping/development processing unit 50 clips an image from the imaging data generated in step S400 using the recording direction and angle of view information determined in step S300, and performs development processing of that range. Execute recording area development processing. Details of the recording area development processing will be described later with reference to FIG. 7E.

In step S600, the primary recording unit 60 (image recording means) executes primary recording processing in which the image developed in step S500 is stored in the primary memory 103 as image data. Details of the primary recording process will be described later with reference to FIG.

In step S700, the transmission unit 70 executes a transfer process to the display device 800 for wirelessly transmitting the video primarily recorded in step S600 to the display device 800 at a designated timing. Details of the transfer processing to the display device 800 will be described later with reference to FIG. 16 .

The steps after step S<b>800 are executed by display device 800 .

In step S800, the display device control unit 801 executes optical correction processing for correcting optical aberration of the image transferred from the camera body 1 in step S700. Details of the optical correction processing will be described later with reference to FIG.

In step S900, the display device control unit 801 performs image stabilizing processing on the image optically corrected in step S800. Details of the anti-vibration processing will be described later with reference to FIG. 19 .

Note that the order of steps S800 and S900 may be reversed. In other words, image stabilization processing may be performed first, and then optical correction may be performed.

In step S1000, the display device control unit 801 (moving image recording means) performs secondary recording to record the image for which the optical correction processing and image stabilization processing in steps S800 and S900 have been completed in the large-capacity nonvolatile memory 814. exit.

Next, with reference to FIGS. 7B to 7F, the subroutine of each step described with reference to FIG. 7A will be described in detail along with the order of processing and other figures.

FIG. 7B is a flowchart of a subroutine of the preparatory operation process in step S100 of FIG. 7A. This process will be described below using the parts shown in FIGS. 2 and 5. FIG.

In step S101, it is determined whether the power switch 11 is ON. If the power remains OFF, it waits, and if it turns ON, the process proceeds to step S102.

In step S102, the mode selected by the imaging mode switch 12 is determined. As a result of determination, if the mode selected by the imaging mode switch 12 is the moving image mode, the process proceeds to step S103.

In step S103, various settings of the moving image mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, after which the process proceeds to step S104. Here, the various settings of the moving image mode include the angle of view setting value V (which is preset to 90° in this embodiment) and the image stabilizing level designated as "strong", "medium", "off", and the like.

In step S104, after starting the operation of the imaging driver 41 for moving image mode, this subroutine is exited.

If the result of determination in step S102 is that the mode selected by the imaging mode switch 12 is the still image mode, the process proceeds to step S106.

In step S106, various settings of the still image mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, after which the process proceeds to step S107. Here, the various settings of the still image mode include the angle of view setting value V (which is preset to 45° in this embodiment) and the vibration reduction level specified by "strong", "medium", "off", etc. .

In step S107, after starting the operation of the imaging driver 41 for the still image mode, this subroutine is exited.

If the result of determination in step S102 is that the mode selected by the imaging mode switch 12 is the preset mode, the process proceeds to step S108. Here, the preset mode is a mode in which an imaging mode is set for the camera body 1 from an external device such as the display device 800, and is one of three imaging modes that can be switched by the imaging mode switch 12. . The preset mode is a mode for custom shooting. Since the camera body 1 is a small wearable device, the camera body 1 is not provided with operation switches or setting screens for changing detailed settings. The detailed settings of the main body 1 are changed.

For example, consider a case where it is desired to continuously capture the same moving image at a field angle of 90° and a field angle of 110°. Since the angle of view is set to 90° in the normal moving image mode, when performing such image capturing, first, after image capturing in the normal moving image mode, the moving image capturing is finished once, and the display device 800 displays the image of the camera body 1. It is necessary to display the setting screen and switch the angle of view to 110°. However, such an operation on the display device 800 is troublesome during some event.

On the other hand, if the preset mode is previously set to a mode for capturing a moving image with an angle of view of 110°, after capturing a moving image with an angle of view of 90°, simply slide the imaging mode switch 12 to "Pre". , the angle of view of 110° can be changed immediately. That is, the user does not need to interrupt the current action and perform the above-described troublesome operations.

In addition, the settings in the preset mode include not only the angle of view, but also the image stabilization level specified by "strong", "medium", "off", etc., and voice recognition settings that are not explained in this embodiment. good too.

In step S108, various settings of the preset mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, after which the process proceeds to step S109. Various settings in the preset mode include a field angle setting value V and image stabilizing levels designated as "strong," "medium," and "off."

In step S109, after starting the operation of the imaging driver 41 for the preset mode, this subroutine is exited.

Here, various settings of the moving image mode read in step S103 will be described with reference to FIG.

FIG. 13 is a diagram showing a menu screen for various settings of the moving image mode, which is displayed on the display unit 803 of the display device 800 before imaging by the camera body 1. As shown in FIG. Note that the same reference numerals are used for the same portions as those in FIG. 1D, and the description thereof is omitted. Note that the display unit 803 has a touch panel function, and the following description will be made on the assumption that it functions by touch operations including operations such as swiping.

13, the menu screen includes a preview screen 831, a zoom lever 832, a recording start/stop button 833, a switch 834, a remaining battery level display 835, a button 836, a lever 837, and an icon display section 838.

On the preview screen 831, it is possible to confirm the image captured by the camera body 1, and it is possible to confirm the zoom amount and the angle of view.

A zoom lever 832 is an operation unit that allows zoom setting by shifting left and right. In this embodiment, four values of 45°, 90°, 110°, and 130° can be set as the angle of view setting value V, but values other than these can be set as the angle of view setting value V by the zoom lever 832. You may make it possible.

The recording start/stop button 833 is a toggle switch having both the functions of the start switch 14 and the stop switch 15 .

A switch 834 is a switch for switching between "OFF" and "ON" of anti-vibration.

The remaining battery level display 835 displays the remaining battery level of the camera body 1 .

A button 836 is a button for changing modes.

A lever 837 is a lever for setting an anti-vibration level. In this embodiment, only "strong" and "medium" can be set as the image stabilization level, but other image stabilization levels such as "weak" may also be set. Alternatively, the image stabilizing level may be set steplessly.

The icon display portion 838 displays a plurality of thumbnail icons for preview.

FIG. 7C is a flowchart of a subroutine of face direction detection processing in step S200 of FIG. 7A. Before describing the details of this process, a face direction detection method using infrared light projection will be described with reference to FIGS. 8A to 8K.

8A is a diagram showing a visible light image of the user's face viewed from the position of the face direction detection window 13. FIG.

The image of FIG. 8A is obtained when the face direction detection window 13 does not have a visible light cut filter component, the visible light is sufficiently transmitted, and the infrared detection processing device 27 is a visible light imaging device. It is the same as the image captured by the device.

The image of FIG. 8A shows the user's face 204 including the neck front 201 above the collarbone, the base of the chin 202, the tip of the chin 203, and the nose.

FIG. 8B is a diagram showing a case where the fluorescent lamp 205 in the room is reflected as a background in the visible light image of the user shown in FIG. 8A.

The visible light image of FIG. 8B shows a plurality of fluorescent lights 205 surrounding the user. As described above, since various backgrounds and the like appear in the image of the user depending on the usage conditions, it becomes difficult for the face direction detection unit 20 and the overall control CPU 101 to separate the image of the face from the visible light image. On the other hand, there is a technique for segmenting such images by using AI or the like, but it requires a high performance of the general control CPU 101 and is not suitable for the camera body 1 which is a portable device.

Therefore, the camera 1 of Example 1 detects the user's face using an infrared light image. Since the face direction detection window 13 is composed of a visible light cut filter, almost no visible light passes through it, so the image of the infrared detection processing device 27 does not become the image shown in FIGS. 8A and 8B.

FIG. 8C shows the case where the user shown in FIG. 8B and the fluorescent lamp as the background thereof are imaged by the sensor of the infrared detection processing device 27 through the face direction detection window 13 in a state where the infrared LED 22 is not turned on. 1 is a diagram showing an infrared image of .

In the infrared image of FIG. 8C, the user's neck and chin are dark. On the other hand, the fluorescent lamp 205 appears rather bright because it has not only visible rays but also infrared rays.

In FIG. 8D, the user shown in FIG. 8B and the fluorescent lamp as the background thereof are imaged by the sensor of the infrared detection processing device 27 through the face direction detection window 13 with the infrared LED 22 turned on. It is a figure which shows the image|video in a case.

In the image of FIG. 8D, the user's neck and chin are brightened. On the other hand, unlike FIG. 8C, the brightness around the fluorescent lamp 205 does not change.

FIG. 8E illustrates a difference image calculated by subtracting the image of FIG. 8C from the image of FIG. 8D. It can be seen that the user's face is highlighted.

In this way, the overall control CPU 101 (image acquisition means) calculates the difference between the images formed by the sensor of the infrared detection processing device 27 when the infrared LED 22 is lit and when the infrared LED 22 is extinguished. A difference image (hereinafter also referred to as a face image) is obtained from which the face of the person is extracted.

The face direction detection unit 20 of this embodiment employs a method of obtaining a face image by extracting the infrared reflection intensity as a two-dimensional image by the infrared detection processing device 27 . The sensor of the infrared detection processing device 27 has a structure similar to that of a general imaging device, and acquires a face image frame by frame. A vertical synchronizing signal (hereinafter referred to as a V signal) for synchronizing the frame is generated by the infrared detection processor 27 and output to the overall control CPU 101 .

FIG. 9 is a timing chart showing the timing of turning on/off the infrared LED 22 and related signals.

FIG. 9(a) shows the timing at which the V signal is generated by the infrared detection processor 27. FIG. When the V signal becomes Hi, the timing of frame synchronization and turning on/off of the infrared LED 22 is measured.

In FIG. 9A, t1 indicates the first facial image acquisition period, and t2 indicates the second facial image acquisition period. FIGS. 9A, 9B, 9C, and 9D are drawn so that the time axes of the horizontal axes are the same.

In FIG. 9B, the vertical axis represents the H position of the image signal output from the sensor of the infrared detection processing device 27 . The infrared detection processor 27 controls the movement of the sensor so that the H position of the image signal is synchronized with the V signal, as shown in FIG. 9(b). As described above, the sensor of the infrared detection processing device 27 employs a structure similar to that of a general imaging device, and its movement is known, so detailed control will be omitted.

FIG. 9(c) shows the switching timing of the IR-ON signal output from the overall control CPU 101 to the infrared LED lighting circuit 21 between Hi and Low. The switching of the IR-ON signal between Hi and Low is controlled by the general control CPU 101 so as to synchronize with the V signal as shown in FIG. 9(c). Specifically, the overall control CPU 101 outputs a Low IR-ON signal to the infrared LED lighting circuit 21 during the period t1, and outputs a High IR-ON signal to the infrared LED lighting circuit 21 during the period t2. Output to circuit 21 .

Here, while the IR-ON signal is Hi, the infrared LED lighting circuit 21 lights the infrared LED 22, and the infrared ray 23 is projected to the user. On the other hand, while the IR-ON signal is Low, the infrared LED lighting circuit 21 turns off the infrared LED 22 .

FIG. 9(d) shows imaging data output from the sensor of the infrared detection processing device 27 to the overall control CPU 101. FIG. The vertical direction indicates the signal intensity, which indicates the amount of light received by the reflected light beam 25 . In other words, during the period t1, the infrared LED 22 is turned off, so there is no reflected light 25 from the user's face, and image data such as that shown in FIG. 8C is obtained. On the other hand, during the period t2, since the infrared LED 22 is lit, there is a reflected light beam 25 from the user's face, and imaging data as shown in FIG. 8D is obtained. Therefore, as shown in FIG. 9(d), the signal intensity during the period t2 is higher than the signal intensity during the period t1 by the reflected light beam 25 from the user's face.

FIG. 9(e) is obtained by subtracting the imaged data during the period t1 from the imaged data during the period t2 in FIG. 9(d). face image data from which only the component of is extracted is obtained.

FIG. 7C shows face direction detection processing in step S200 including the operations described above with reference to FIGS. 8C to 8E and FIG.

First, in step S201, when the V signal output from the infrared detection processing device 27 becomes Hi, the timing V1 at which the period t1 starts is acquired. When the timing V1 is obtained, the process proceeds to step S202.

Then, in step S202, the IR-ON signal is set to Low and output to the infrared LED lighting circuit 21. FIG. Thereby, the infrared LED 22 is extinguished.

In step S203, one frame of imaging data output from the infrared detection processing device 27 during the period t1 is read, and the data is temporarily stored in the primary memory 103 as Frame1.

In step S204, when the V signal output from the infrared detection processing device 27 reaches the timing V2 when the period t2 starts, the process proceeds to step S205.

At step S 205 , the IR-ON signal is set to Hi and output to the infrared LED lighting circuit 21 . Thereby, the infrared LED 22 is lit.

In step S206, one frame of imaging data output from the infrared detection processing device 27 during the period t2 is read, and the data is temporarily stored in the primary memory 103 as Frame2.

At step S 207 , the IR-ON signal is set to Low and output to the infrared LED lighting circuit 21 . As a result, the infrared LED 22 is extinguished.

In step S208, Frame1 and Frame2 are read out from the primary memory 103, and the difference obtained by subtracting Frame1 from Frame2, and the light intensity Fn of the 25 reflected light rays of the user corresponding to the facial image in FIG. This is a process commonly called blacking.)

In step S209, the neck position (neck rotation center) is extracted from the light intensity Fn.

First, the overall control CPU 101 (dividing means) divides the face image into a plurality of distance areas, which will be explained with reference to FIG. 8F, based on the light intensity Fn.

FIG. 8F shows the distribution of light intensity for each part of the user's face and neck. FIG. 10 is a diagram showing a case where the scale is adjusted according to the intensity;

FIG. 8F(a) is a diagram showing the distribution of the light intensity of the reflected light ray 25 in the face image of FIG. 8E divided into regions and shown in gray stages. For the sake of explanation, the Xf axis is taken in the direction from the center of the user's neck to the tip of the chin.

In FIG. 8F(a), the horizontal axis indicates the light intensity on the Xf axis in FIG. 8F(a), and the vertical axis indicates the Xf axis. The horizontal axis indicates higher light intensity toward the right.

In FIG. 8F(a), the face image is divided into six areas (distance areas) 211 to 216 according to the light intensity.

Region 211 is the region with the highest light intensity and is shown in white as shades of gray.

Region 212 is a region of slightly less light intensity than region 211 and is shown in a much lighter gray color as a gray scale.

Region 213 is a region of even lower light intensity than region 212 and is shown in light gray as a gray scale.

Region 214 is a region of even lower light intensity than region 213 and is shown in intermediate gray as a gray scale.

Area 215 is an area where the light intensity is even lower than that of area 214, and is shown in slightly darker gray as a gray scale.

Area 216 is the area with the lowest light intensity and is the darkest shade of gray. The area above the area 216 is black with no light intensity.

This light intensity will be explained in detail below with reference to FIG.

FIG. 10 is a diagram for explaining vertical movement of the user's face, and shows a state observed from the left lateral direction of the user.

FIG. 10(a) is a diagram showing a state in which the user faces the front. An imaging/detecting unit 10 is located in front of the user's clavicle. Infrared rays 23 from an infrared LED 22 are emitted to the lower portion of the user's head from the face direction detection window 13 in the upper portion of the photographing/detecting section 10 . Dn is the distance from the face direction detection window 13 to the neck 200 on the collarbone of the user, Db is the distance from the face direction detection window 13 to the base of the chin 202, and Db is the distance from the face direction detection window 13 to the chin 203. Assuming Dc, it can be seen that the distance increases in the order of Dn, Db, and Dc. Since the light intensity is inversely proportional to the square of the distance, the light intensity when the reflected light beam 25 from the infrared irradiation surface 24 is imaged on the sensor of the infrared detection processing device 27 is the neck 200, the chin base 202, It becomes weaker in order of the tip of the chin 203 . Further, it can be seen that the light intensity of the face 204 including the nose, which is farther from the face direction detection window 13 than Dc, is even darker. That is, in the case shown in FIG. 10A, an image having the light intensity distribution shown in FIG. 8F is acquired.

The configuration of the face direction detection unit 20 is not limited to the configuration shown in this embodiment as long as the face direction of the user can be detected. For example, an infrared pattern may be emitted from the infrared LED 22 (infrared pattern irradiation means), and the infrared pattern reflected from the irradiation target may be detected by the sensor (infrared pattern detection means) of the infrared detection processing device 27. good. In this case, the sensor of the infrared detection processing device 27 is preferably a structured light sensor. Further, the sensor of the infrared detection processing device 27 may be a sensor (infrared phase comparison means) that compares the phases of the infrared rays 23 and the reflected light beam 25, such as a Tof sensor.

Next, extraction of the neck position in step S209 of FIG. 7C will be described with reference to FIG. 8G.

FIG. 8G(a) is a diagram in which the symbols indicating each part of the user's body in FIG. is.

A white region 211 corresponds to the neck 200 (FIG. 10(a)), and a fairly light gray region 212 corresponds to the neck front 201 (FIG. 10(a)), and is light gray. A colored region 213 corresponds to the base of the chin 202 (FIG. 10(a)). A middle gray area 214 corresponds to the tip of the chin 203 (FIG. 10(a)), and a slightly darker gray area 215 is located below the face 204 (FIG. 10(a)). It corresponds to the lower part of the face around the lips and the periphery. In addition, a darker gray area 216 corresponds to the nose located in the center of the face 204 (FIG. 10(a)) and the surrounding upper part of the face.

As shown in FIG. 10A, the distances Db and Dc are less different than the distances from the face direction detection window 13 to other parts of the user. There is also little difference in reflected light intensity in the gray area 214 of .

On the other hand, as shown in FIG. 10A, among the distances from the face direction detection window 13 to each part of the user, the distance Dn is the shortest distance. A point 211 has the highest reflection intensity.

Therefore, the overall control CPU 101 (setting means) determines that the area 211 is around the neck 200, is the center of the left and right of the area 211, and is the closest to the photographing/detecting unit 10, indicated by a double circle in FIG. 8G(a). A position 206 is set at the center of neck rotation (hereinafter referred to as neck position 206). The processing up to this point is the content performed in step S209 of FIG. 7C.

Next, extraction of the tip of the chin position in step S210 of FIG. 7C will be described with reference to FIG. 8G.

A medium gray area 214 that is lighter than the area 215 corresponding to the lower part of the face including the lips in the face 204 shown in FIG. 8G(a) is the area including the tip of the chin. As can be seen from FIG. 8G(a), the rate of change in the distance from the face direction detection window 13 increases, so the light intensity drops sharply in the area 215 that is in contact with the area 214 . The overall control CPU 101 determines that a brighter area 214 adjacent to the area 215 where the light intensity sharply drops is the chin area. Furthermore, the overall control CPU 101 calculates (extracts) the position 207 of the tip of the chin at the center of the area 214 in the left and right direction and farthest from the neck position 206 (the position indicated by the black circle in FIG. 8G(a)).

For example, FIGS. 8H and 8I show changes when the face is turned to the right.

FIG. 8H is a diagram showing a difference image calculated in the same manner as in FIG. 8E when the user's face is facing right. In FIG. 8I, the shading of the difference image in FIG. 8H is adjusted by adjusting the scale to the light intensity of the reflected infrared rays projected onto the user's face/neck, and the neck position 206, which is the position of the neck rotation center, is adjusted. is a diagram in which a double circle indicating a position 207r and a black circle indicating a chin tip position 207r are superimposed.

Since the user's face has turned to the right, area 214 moves to area 214r shown in FIG. A region 215 corresponding to the lower part of the face including the lips in the face 204 also moves to a leftward region 215r when viewed from the imaging/detecting unit 10 side.

Therefore, the overall control CPU 101 determines that the area 214r in front of 215r where the light intensity drops sharply is the chin area. Furthermore, the overall control CPU 101 calculates (extracts) the position of the left and right center of 214r and the farthest position from the neck position 206 (the position indicated by the black circle in FIG. 8I) as the chin position 207r.

After that, the overall control CPU 101 obtains a movement angle θr indicating how much the chin position 207r in FIG. 8I has moved from the chin position 207 in FIG. 8G(a) to the right around the neck position 206. As shown in FIG. 8I, the movement angle θr is the angle of the user's face in the horizontal direction.

By the above method, in step S210, the lateral face angle of the user is calculated from the chin position detected by the infrared detection processing device 27 of the face direction detection unit 20 (three-dimensional detection sensor).

Next, the detection of faces directed upward will be described.

FIG. 10(b) is a diagram showing a state in which the user faces horizontally, and FIG. 10(c) is a diagram showing a state in which the user faces upward 33° from the horizontal direction. is.

In FIG. 10B, the distance from the face direction detection window 13 to the chin 203 is Ffh, and in FIG. 10C, the distance from the face direction detection window 13 to the chin 203u is Ffu.

As shown in FIG. 10(c), since the chin 203u moves upward together with the face, it can be seen that Ffu is longer than Ffh.

FIG. 8J is a diagram showing an image of the user seen through the face direction detection window 13 when the user faces 33° above the horizontal. As shown in FIG. 10(c), since the user is facing upward, the face 204 including lips and nose cannot be seen from the face direction detection window 13 located under the user's chin. I can see up to 203 ahead. FIG. 8K shows the light intensity distribution of the reflected light beam 25 when the infrared ray 23 is applied to the user in the state of FIG. 10(c). In FIG. 8K (a), the gradation of the difference image calculated by the same method as in FIG. It is a diagram in which a double circle indicating the original position 206 and a black circle indicating the chin tip position 207u are superimposed. Graphs (a) and (c) of FIG. 8K show changes in density of the left image, (a) corresponds to the graph of FIG. 8F, and (c) corresponds to the graph of FIG. 8G.

The six regions 211u to 216u corresponding to the light intensities in FIG. 8K(a) are regions with the same light intensity as the regions shown in FIG. 8F, with "u" added. The light intensity of the user's chin 203 was in the middle gray area 214 in FIG. 8F(a), but shifted to the gray side in FIG. It can be seen that there is Thus, as shown in FIG. 10(c), as a result of Ffu being longer than Ffh, the light intensity of the reflected light beam 25 from the tip of the user's chin 203 is weakened in inverse proportion to the square of the distance. , can be detected by the infrared detection processor 27 .

Next, the detection of faces facing downward will be described.

FIG. 10(d) is a diagram showing a state in which the face of the user is directed downward by 22° from the horizontal direction.

In FIG. 10D, the distance from the face direction detection window 13 to the tip of the chin 203d is Ffd.

As shown in FIG. 10(d), since the chin 203d moves downward together with the face, Ffd is shorter than Ffh, and the light intensity of the reflected light ray 25 from the chin 203 increases.

Returning to FIG. 7C, in step S211, the overall control CPU 101 (distance calculation means) determines the position of the chin from the light intensity of the chin position detected by the infrared detection processing device 27 of the face direction detection unit 20 (three-dimensional detection sensor). A distance from the forward position to the face direction detection window 13 is calculated. Based on this, the angle of the face in the vertical direction is also calculated.

In step S212, the overall control CPU 101 converts the angle of the horizontal direction (first detection direction) of the face obtained in steps S210 and S211 and the vertical direction (second detection direction) perpendicular thereto to the user's three angles. It is stored in the primary memory 103 as the dimensional viewing direction vi (i is an arbitrary code). For example, the observation direction vo when the user is observing the center of the front is 0° in the horizontal direction θh and 0° in the vertical direction θv, so the vector information is [0°, 0°]. Also, the observation direction vr when the user is observing 45 degrees to the right becomes vector information of [45 degrees, 0 degrees].

Although the vertical angle of the face is calculated by detecting the distance from the face direction detection window 13 in step S211, the method is not limited to this method. For example, the angle change may be calculated by comparing the variation level of the light intensity of the chin 203 . That is, the gradient CDh of the reflected light intensity from the chin root 202 to the tip of the chin 203 in FIG. 8G(a) is compared with the gradient CDu of the reflected light intensity from the chin root 202 to the tip of the chin 203 in FIG. 8K(c). , the angular change of the face may be calculated.

FIG. 7D is a flowchart of a subroutine of the recording direction/range determination process in step S300 of FIG. 7A. Before describing the details of this process, first, using FIG. 11A, a super-wide-angle image for which the recording direction and recording range in this embodiment are determined will be described.

In the camera body 1 of the present embodiment, the photographing unit 40 uses the ultra-wide-angle imaging lens 16 to capture a super-wide-angle image around the photographing/detecting unit 10, and a part of the image is cut out to obtain an image in the observation direction. have achieved

FIG. 11A is a diagram showing the target field of view 125 in the ultra-wide-angle image captured by the imaging unit 40 when the user faces the front.

As shown in FIG. 11A, the imageable pixel area 121 of the solid-state imaging device 42 is a rectangular area. The effective projection area 122 (predetermined area) is an area of a circular half-celestial sphere image projected onto the solid-state imaging device 42 by the imaging lens 16 as a fisheye. The imaging lens 16 is adjusted so that the center of the pixel region 121 and the center of the effective projection portion 122 are aligned.

The outermost periphery of the circular effective projection portion 122 indicates a position with an FOV (Field of view) angle of 180°. When the user is looking at the horizontal and vertical center, the angular range of the target field of view 125, which is the area to be imaged and recorded, is 90° (half the VOF angle) around the center of the effective projection section 122. FIG. The imaging lens 16 of this embodiment can also introduce light rays outside the effective projection portion 122, and can project light rays up to a maximum FOV angle of 192° onto the solid-state image pickup device 42 as a fisheye. However, when the effective projection area 122 is exceeded, the optical performance is greatly deteriorated, such as the resolving power being extremely reduced, the amount of light being reduced, and the distortion becoming stronger. Therefore, in this embodiment, the recording area is an example in which the image in the observation direction is cut out only from the image projected on the pixel area 121 (hereinafter simply referred to as the ultra-wide-angle image) of the half-celestial image displayed on the effective projection unit 122. will explain.

In this embodiment, since the size of the effective projection portion 122 in the vertical direction is larger than the size of the short side of the pixel region 121, the images at the upper and lower ends of the effective projection portion 122 are out of the pixel region 121, but the present invention is not limited to this. . For example, the configuration of the imaging lens 16 may be changed to design the optical system so that the entire effective projection section 122 can be accommodated within the pixel area 121 .

An invalid pixel area 123 is a pixel area that is not included in the effective projection area 122 in the pixel area 121 .

The target field of view 125 is an area indicating the range of cutting out the image in the viewing direction of the user from the ultra-wide-angle image, and is defined by a preset left, right, up, and down angle of view centered on the viewing direction (here, 45°, FOV angle of 90°). ). In the example of FIG. 11A, since the user faces the front, the center of the target field of view 125 is the observation direction vo, which is the center of the effective projection portion 122 .

The ultra-wide-angle image shown in FIG. 11A includes a subject A131 that is a child, a subject B132 that is a staircase that the child who is subject A is trying to climb, and a subject C133 that is a locomotive-shaped playground equipment.

Next, FIG. 7D shows the recording direction/range determination processing in step S300 that is executed to obtain the image in the viewing direction from the super-wide-angle image described using FIG. 11A. This process will be described below with reference to FIGS.

In step S<b>301 , the preset view angle setting value V is read out from the primary memory 103 to acquire it.

In the present embodiment, all the view angles, 45°, 90°, 110°, and 130°, at which the image clipping/development processing unit 50 can clip the video in the viewing direction from the super-wide-angle image, are the view angle setting values V is stored in the built-in nonvolatile memory 102 as . Also, in any one of steps S103, S106, and S108 in FIG. there is

In step S301, the observation direction vi determined in step S212 is determined as the recording direction, and the image of the target field of view 125 cut out from the ultra-wide-angle image with the above-obtained view angle setting value V centered on this is processed as a primary image. Save in memory 103 .

For example, when the view angle setting value V is 90° and the observation direction vo (vector information [0°, 0°]) is detected in the face direction detection process (FIG. 7C), the center O of the effective projection unit 122 is set to the target field of view 125 (FIG. 11A) in which the angular range in the left, right, and up and down directions is 90°. FIG. 11B is a diagram showing an image of the target field of view 125 cut out from the ultra-wide-angle image of FIG. 11A. That is, the overall control CPU 101 (relative position setting means) sets the angle of the face direction detected by the face direction detection unit 20 to the observation direction vi, which is vector information indicating the relative position of the target field of view 125 with respect to the ultra-wide-angle image. ing.

Here, in the case of the observation direction vo, since the influence of the optical distortion by the imaging lens 16 can be almost ignored, the shape of the set target field of view 125 can be used as it is for the target field of view 125o (FIG. 12A) after distortion conversion in step S303, which will be described later. shape. Hereinafter, the target field of view 125 after distortion conversion in the case of the observation direction vi is referred to as a target field of view 125i.

Next, in step S302, a preset anti-vibration level is read out from the primary memory 103 to obtain it.

In this embodiment, as described above, the anti-vibration level included in the various setting values read out from the built-in non-volatile memory 102 is set in one of steps S103, S106, and S108, and stored in the primary memory 103. ing.

Further, in step S302, the number of spare pixels Pis for image stabilization is set based on the obtained image stabilization level.

In the anti-vibration processing, an image that follows the image in the direction opposite to the direction of the blur is obtained by following the blur amount of the photographing/detecting unit 10 . For this reason, in this embodiment, a preliminary area for image stabilization necessary for image stabilization is provided around the target visual field 125i.

Further, in this embodiment, the built-in non-volatile memory 102 stores a table that holds the value of the number of anti-shake pixels Pis associated with each anti-shake level. For example, if the vibration reduction level is "medium", a vibration correction preliminary area with a width of 100 pixels, which is the number of vibration correction preliminary pixels Pis read from the table, is set surrounding the target field of view.

FIG. 12E is a diagram showing an example in which an anti-vibration preliminary area corresponding to a predetermined anti-vibration level is provided around the target visual field 125o shown in FIG. 12A. Here, the case where the image stabilization level is "medium", that is, the number of image stabilization spare pixels Pis is 100 pixels will be described.

As indicated by the dotted line in FIG. 12E, anti-vibration spare pixel frames 126o each having a width of 100 pixels, which is the number of anti-vibration spare pixels Pis, are set on the top, bottom, left, and right of the target field of view 125o.

For simplicity of explanation, FIGS. 12A and 12E have explained the case where the observation direction vi coincides with the center O of the effective projection section 122 (the optical axis center of the imaging lens 16). On the other hand, when the observation direction vi is the periphery of the effective projection portion 122, conversion is required to reduce the effects of optical distortion.

In step S303, the shape of the target field of view 125 set in step S301 is corrected (distorted) in consideration of the observation direction vi and the optical characteristics of the imaging lens 16 to generate a target field of view 125i. Similarly, the number of spare pixels for vibration reduction Pis set in step S302 is also corrected in consideration of the viewing direction vi and the optical characteristics of the imaging lens 16. FIG.

For example, assume that the angle of view setting value V is 90° and the user is observing 45° to the right of the center o. In this case, the observation direction vr (vector information [45°, 0°]) is determined in step S212, and the target field of view 125 is a range of 45° left and right and 45° up and down centered on the observation direction vr. However, considering the optical characteristics of the imaging lens 16, the target field of view 125 is corrected to the target field of view 125r shown in FIG. 12B.

As shown in FIG. 12B, the target field of view 125r widens toward the periphery of the effective projection portion 122, and the position of the viewing direction vr is slightly inside the center of the target field of view 125r. This is because, in this embodiment, the imaging lens 16 has an optical design similar to a stereographic fisheye. Incidentally, if the imaging lens 16 is designed with an equidistant projection fisheye, an equisolid angle projection fisheye, or an orthogonal projection fisheye, etc., the relationship will change. performed for

FIG. 12F is a diagram showing an example in which an anti-vibration preliminary area 126r corresponding to the same anti-vibration level "middle" as the anti-vibration preliminary area in FIG. 12E is provided around the target visual field 125r shown in FIG. 12B.

In the anti-vibration preliminary area 126o (FIG. 12E), a width of 100 pixels, which is the number of anti-vibration preliminary pixels Pis, is set in each of the top, bottom, left, and right of the target visual field 125o. On the other hand, in the anti-vibration preliminary area 126r (FIG. 12F), the number of anti-vibration preliminary pixels Pis is corrected and increased toward the periphery of the effective projection portion 122. FIG.

As described above, the shape of the image stabilization preliminary area required for image stabilization provided around the target field of view 125r is also the same as the shape of the image stabilization preliminary area 126r shown in FIG. 12F. The amount of correction increases toward the periphery of . This is also because, in this embodiment, the imaging lens 16 has an optical design similar to a stereoscopic projection fisheye. If the imaging lens 16 is designed with an equidistant projection fisheye, an equisolid angle projection fisheye, orthographic projection fisheye, etc., the relationship will change. 126r is corrected.

The process of sequentially switching the shapes of the target field of view 125 and its anti-vibration preliminary area in consideration of the optical characteristics of the imaging lens 16, which is executed in step S303, is a complicated process. Therefore, in this embodiment, the processing of step S303 is executed using a table in the built-in nonvolatile memory 102 that holds the shape of the target field of view 125i for each viewing direction vi and the shape of the preliminary area for image stabilization. . Incidentally, depending on the optical design of the imaging lens 16 mentioned above, an arithmetic expression may be stored in the overall control CPU 101, and the optical distortion value may be calculated by the arithmetic expression.

In step S304, the position and size of the video recording frame are calculated.

As described above, the preliminary region 126i for image stabilization required for image stabilization is provided around the target visual field 125i. However, when the position in the viewing direction vi approaches the periphery of the effective projection portion 122, the shape becomes quite special, such as the preliminary region 126r for vibration isolation.

The overall control CPU 101 can cut out only the image of such a special shape and perform development processing. However, it is not common to use a non-rectangular image when recording as image data in step S600 or transferring to display device 800 in step S700. Therefore, in step S304, the position and size of a rectangular image recording frame 127i that includes the entire anti-vibration preliminary area 126i are calculated.

FIG. 12F shows a video recording frame 127r indicated by a dashed line, calculated in step S304 for the anti-shake preliminary area 126r.

In step S305, the position and size of the video recording frame 127i calculated in step S304 are recorded in the primary memory 103. FIG.

In this embodiment, the upper left coordinates Xi, Yi of the image recording frame 127i in the ultra-wide-angle image are recorded as the position of the image recording frame 127i, and the horizontal width WXi and the vertical width WXi of the image recording frame 127i from the coordinates Xi, Yi are recorded. The width WYi is recorded as the size of the video recording frame 127i. For example, for the video recording frame 127r shown in FIG. 12F, the illustrated coordinates Xr, Yr, horizontal width WXr, and vertical width WYr are recorded in step S305. The coordinates Xi and Yi are XY coordinates with a predetermined reference point, specifically, the optical center of the imaging lens 16 as the origin.

When the anti-vibration spare area 126i and the video recording frame 127i are determined in this manner, the subroutine shown in FIG. 7D is exited.

In the description so far, for the sake of simplification of the explanation of the complicated optical distortion conversion, the observation direction vi is taken as an example of the observation direction including the horizontal 0°, that is, the observation direction vo (vector information [0°, 0°] ) and observation direction vr (vector information [45°, 0°]). However, in practice, the viewing direction vi of the user varies. Therefore, the recording area development process executed when the horizontal angle is not 0° will be described below.

For example, the target visual field 125m when the view angle setting value V is 90° and the observation direction is vm [−42°, −40°] is as shown in FIG. 12C.

Also, even if the observation direction vm (vector information [-42°, -40°]) is the same as the target field of view 125m, if the angle of view setting value V is 45°, the target field of view 125m The target field of view is 128 m, which is one size smaller. Furthermore, for the target field of view 128m, a preliminary region 129m for image stabilization and a frame 130m for video recording are set as shown in FIG. 12G.

Step S400 is a basic operation of imaging, and since a general sequence of the imaging unit 40 is used, detailed description will be omitted. In this embodiment, the imaging signal processing circuit 43 in the imaging unit 40 converts the signal output from the solid-state imaging device 42 in a specific output format (examples of standards: MIPI, SLVS) to a general sensor. It also performs a process of correcting the imaging data to read-out method.

When the mode selected by the imaging mode switch 12 is the moving image mode, the imaging unit 40 starts recording when the start switch 14 is pressed. After that, when the stop switch 15 is pressed, the recording ends. On the other hand, when the mode selected by the imaging mode switch 12 is the still image mode, the imaging unit 40 takes a still image each time the start switch 14 is pressed.

FIG. 7E is a flowchart of a subroutine of recording area development processing in step S500 of FIG. 7A.

In step S501, raw data of the entire area of the imaging data (ultra-wide-angle video) generated by the imaging unit 40 in step S400 is obtained and input to an image capture unit called a head unit (not shown) of the overall control CPU 101.

Next, in step S502, based on the coordinates Xi, Yi, width WXi, and height WYi recorded in the primary memory 103 in step S305, a portion of the video recording frame 127i is cut out from the ultra-wide-angle video acquired in step S501. After this cropping, the crop development process (FIG. 7F) consisting of steps S503 to S508 executed below is started only for the pixels in the anti-vibration preliminary area 126i. As a result, the amount of calculation can be greatly reduced compared to the case where development processing is performed on the entire area of the super-wide-angle image read in step S501, and the calculation time and power can be reduced.

As shown in FIG. 7F, when the mode selected by the imaging mode switch 12 is the moving image mode, the processes of steps S200 and S300 and the process of step S400 are executed in parallel at the same or different frame rates. be done. That is, each time the raw data of the entire area for one frame generated by the imaging unit 40 is acquired, cropping is performed based on the coordinates Xi and Yi, the horizontal width WXi, and the vertical width WYi recorded in the primary memory 103 at that time. Development processing is performed.

When the crop development process for the pixels in the anti-vibration preliminary area 126i is started, first, in step S503, color interpolation is performed to complement the color pixel information arranged in the Bayer array.

Thereafter, after adjusting the white balance in step S504, color conversion is performed in step S505.

In step S506, gamma correction is performed to correct the gradation according to a preset gamma correction value.

In step S507, edge enhancement is performed in accordance with the image size.

In step S508, the data is converted into a data format that can be temporarily stored by performing compression and other processing, and after recording in the primary memory 103, this subroutine is exited. The details of the data format that can be temporarily stored will be described later.

It should be noted that the order of the cropping development processing executed in steps S503 to S508 and the presence or absence of processing may be determined in accordance with the camera system, and are not intended to limit the present invention.

Also, when the moving image mode is selected, the processing from steps S200 to S500 is repeatedly executed until the recording ends.

According to this process, the amount of calculation can be greatly reduced compared to the case where the development process is performed on the entire area read in step S501. Therefore, an inexpensive and low power consumption microcomputer can be used as the overall control CPU 101, heat generation in the overall control CPU 101 can be suppressed, and the battery 94 can last longer.

Further, in this embodiment, in order to lighten the control load of the overall control CPU 101, the image optical correction processing (step S800 in FIG. 7A) and image stabilization processing (step S900 in FIG. 7A) are not performed in the camera body 1, and the display After transferring to the device 800 , the display device control unit 801 performs the processing. Therefore, if only data of an image partially clipped from a projected ultra-wide-angle image is sent to the display device 800, optical correction processing and vibration reduction processing cannot be performed. In other words, the data of the clipped image alone does not have the position information used for substituting it into the formula for the optical correction processing or referring to the correction table for the image stabilizing processing. 800 cannot run correctly. Therefore, in this embodiment, not only data of the clipped image but also correction data including information on the clipping position of the image from the ultra-wide-angle image are transmitted from the camera body 1 to the display device 800 .

If the clipped image is a still image, even if the data of the still image and the correction data are separately transmitted to the display device 800, the data of the still image and the correction data are in one-to-one correspondence. In 800, correct optical correction processing and image stabilizing processing can be performed. On the other hand, if the clipped image is a moving image, if the data of the moving image and the correction data are separately transmitted to the display device 800, it is determined which correction data for each frame of the moving image is the transmitted correction data. becomes difficult. In particular, if the clock rate of the overall control CPU 101 in the camera body 1 and the clock rate of the display device control unit 801 in the display device 800 are slightly different, the difference between the overall control CPU 101 and the display device control unit 801 will occur after several minutes of video shooting. out of sync. As a result, the display device control unit 801 corrects the frame to be processed with correction data different from the correction data corresponding thereto.

Therefore, in the present embodiment, when data of a moving image cut out from the camera body 1 is transmitted to the display device 800, the correction data is appropriately given to the data of the moving image. The method will be described below.

FIG. 14 is a flowchart of a subroutine of primary recording processing in step S600 of FIG. 7A. This processing will be described below with reference to FIG. 15 as well. FIG. 14 shows processing when the mode selected by the imaging mode switch 12 is the moving image mode. If the selected mode is still image mode, this process starts from step S601 and ends when step S606 ends.

In step S601a, the overall control CPU 101 reads out one frame image for which steps S601 to S606 have not been processed from the moving image developed in the recording area development process (FIG. 7E). Also, the overall control CPU 101 (metadata generation means) generates correction data, which is metadata of the read frame.

In step S601, the overall control CPU 101 attaches information about the clipping position of the image of the frame read out in step S600 to the correction data. The information attached here is the coordinates Xi and Yi of the video recording frame 127i acquired in step S305. The information attached here may be vector information indicating the viewing direction Vi.

In step S602, the overall control CPU 101 (optical correction value obtaining means) obtains an optical correction value. The optical correction value is the optical distortion value set in step S303. Alternatively, correction values corresponding to lens optical characteristics such as peripheral light amount correction values and diffraction correction may be used.

In step S603, the overall control CPU 101 attaches the optical correction value used for distortion conversion in step S602 to the correction data.

In step S604, the overall control CPU 101 determines whether or not the mode is anti-vibration mode. Specifically, if the preset anti-shake mode is "medium" or "strong," it is determined that the anti-shake mode is on, and the process proceeds to step S605. On the other hand, if the anti-vibration mode set in advance is "OFF", it is determined that the mode is not anti-vibration mode, and the process proceeds to step S606. The reason why step S605 is skipped when the anti-shake mode is "OFF" is that the amount of data calculated by the overall control CPU 101 and the amount of data during wireless transmission can be reduced by skipping. This is because power consumption and heat generation can also be reduced. Although the reduction of the data used for image stabilizing processing has been described here, the peripheral illumination correction value and the data regarding the presence/absence of diffraction correction included in the optical correction value acquired in step S602 may be reduced.

In this embodiment, the anti-vibration mode is set in advance by the user's operation on the display device 800 , but may be set as an initial setting of the camera body 1 . Also, in the case of a camera system that switches the presence/absence of image stabilizing processing after transfer to the display device 800, step S604 may be omitted and the process may proceed directly from step S603 to step S605.

In step S605, the overall control CPU 101 (movement amount detection means) attaches the anti-vibration mode acquired in step S302 and the gyro data during moving image shooting linked to the frame read out in step S601a in the primary memory 813 to the correction data. .

At step S606, the overall control CPU 101 encodes the image data of the frame read out at step S600 and the correction data to which various data are attached at steps S601 to S605 to encode the video file 1000 (FIG. 15). to update. Note that when the first frame of the moving image is read in step S601a, the image file 1000 is generated in step S606.

In step S607, the overall control CPU 101 determines whether reading of images of all frames of the moving image developed in the recording area development process (FIG. 7E) has been completed. If not, the process returns to step S601a. On the other hand, if it has ended, this subroutine is exited. The generated video file 1000 is saved in the built-in non-volatile memory 102 . It may be stored not only in the primary memory 813 and the built-in non-volatile memory 102 described above, but also in the large-capacity non-volatile memory 51 . Also, a transfer process (step S700 in FIG. 7A) is executed to transfer the generated video file 1000 to the display device 800 immediately. After being transferred to the display device 800 , the video file 1000 may be stored in the primary memory 813 .

Here, in this embodiment, "encoding" refers to combining video data and correction data into a single file. Can be compressed.

FIG. 15 is a diagram showing the data structure of the video file 1000. As shown in FIG.

A video file 1000 is composed of a header 1001 and a frame portion 1002 . The frame portion 1002 is composed of a frame data set BR>, which is a set of images of each frame constituting a moving image and frame metadata corresponding thereto. That is, the frame 1002 portion has frame data sets for the total number of frames of the moving image.

In the present embodiment, the frame metadata is information encoded with correction data to which cutout positions (position information in the video), optical correction values, and gyro data are attached as necessary, but is not limited to this. For example, depending on the imaging mode selected by the imaging mode switch 12, other information is attached to the frame metadata, or information in the frame metadata is deleted, thereby reducing the information amount of the frame metadata. You can change it.

In the header 1001, the offset value or start address of each frame to the frame data set is recorded. Alternatively, the header 1001 may store metadata such as time and size corresponding to the video file 1000 .

In this way, in the primary recording process (FIG. 14), the image file 1000 in which each frame of the moving image developed in the recording area development process (FIG. 7E) and its metadata are set is transferred to the display device 800. be done. Therefore, even if the clock rate of the overall control CPU 101 of the camera body 1 and the clock rate of the display device control section 801 of the display device 800 are slightly different, the display device control section 801 can correct the moving image developed by the camera body 1. process can be performed reliably.

In this embodiment, the optical correction value is included in the frame metadata, but the optical correction value may be applied to the entire image.

FIG. 16 is a flowchart of a subroutine of transfer processing to display device 800 in step S700 of FIG. 7A. FIG. 16 shows processing when the mode selected by the imaging mode switch 12 is the moving image mode. Note that if the selected mode is the still image mode, this process starts from step S702.

In step S701, it is determined whether or not recording of the moving image by the imaging unit 40 (step S400) is completed, or whether recording is in progress. Here, when a moving image is being recorded (moving image is being captured), recording area development processing (step S500) for each frame and updating of the image file 1000 (step S606) in the primary recording processing (step S600) are sequentially performed. It will be in the state of being done. Since wireless transfer has a large power load, if it is performed in parallel with recording, a large battery capacity of the battery 94 is required, or heat generation countermeasures need to be taken separately. Also, from the point of view of computing power, performing wireless transfer in parallel with video recording increases the computing load, so it is necessary to prepare a high-spec overall control CPU 101, which increases the cost. In view of this, in this embodiment, after waiting for the end of recording of the moving image (YES in step S701), the process proceeds to step S702 to establish connection with the display device 800. FIG. However, if the camera system of the present embodiment has sufficient power supplied from the battery 94 and does not require a separate countermeasure against heat generation, the display device 800 may be set in advance when the camera body 1 is activated or before recording is started. You can connect with

In step S 702 , a connection is established with the display device 800 via the high-speed wireless unit 72 in order to transfer the video file 1000 with a large amount of data to the display device 800 . The low-power wireless unit 71 is used for transferring low-resolution video (or video) for checking the angle of view to the display device 800 and for transmitting and receiving various setting values to and from the display device 800. Since it takes time to transfer the video file 1000, it is not used.

In step S703, the video file 1000 is transferred to the display device 800 via the high-speed wireless unit 72, and when the transfer is completed, the process advances to step S704, closes the connection with the display device 800, and exits from this subroutine.

Up to this point, we have explained the case of transferring a single video file containing images of all frames of a single moving image. good. If the video file has the data structure shown in FIG. 15, even if one moving image is transferred to the display device 800 as a plurality of video files, the moving image can be corrected on the display device 800 without timing deviation from the correction data. It becomes possible.

FIG. 17 is a flowchart of a subroutine of optical correction processing in step S800 of FIG. 7A. This processing will be described below with reference to FIG. 18 as well. Note that, as described above, this processing is processing executed by the display device control unit 801 of the display device 800 .

In step S801, first, the display device control section 801 (video file receiving means) receives the video file 1000 from the camera body 1 transferred in the transfer processing to the display device 800 (step S700). After that, the display device control unit 801 (first extraction means) acquires the optical correction values extracted from the received video file 1000 .

Subsequently, in step S<b>802 , the display device control unit 801 (second extraction unit) acquires a video (one frame image obtained by capturing a moving image) from the video file 1000 .

In step S803, the display device control unit 801 (frame image correction means) corrects the optical aberration of the image acquired in step S802 using the optical correction value acquired in step S801, and stores the corrected image in the primary memory 813. . When cutting out the image acquired in step S802 when optical correction is performed, the image is cut out in an image range narrower than the development range (target visual field 125i) determined in step S303 (cutout development region).

FIG. 18 is a diagram for explaining the process of performing distortion aberration correction in step S803 of FIG.

FIG. 18(a) is a diagram showing the position of the subject 1401 as seen by the user's naked eye during imaging, and FIG.

FIG. 18(c) shows the developed area 1402 in the image of FIG. 18(b). Here, the development area 1402 is the clipped development area described above.

FIG. 18(d) is a view showing a clipped developed image in which the image of the development area 1402 is clipped, and FIG. 18(e) is a view showing an image obtained by correcting the distortion of the clipped developed image of FIG. 18(d). is. Since the clipping process is performed during distortion correction of the clipped developed image, the angle of view of the clipped developed image shown in FIG. 18(e) is even smaller than that of the clipped developed image shown in FIG.

FIG. 19 is a flow chart of a subroutine of image stabilizing processing in step S900 of FIG. 7A. This process will be described below with reference to FIG. 18(f) as well. Note that, as described above, this processing is processing executed by the display device control unit 801 of the display device 800 .

In step S901, the display device control unit 801 extracts the gyro data of the current frame and the previous frame from the frame metadata of the video file 1000, and the blur amount Vn _-1 ^Det calculated for the previous frame in step S902, which will be described later. to get After that, an approximate shake amount V _n ^Pre is calculated from these pieces of information. In this embodiment, the current frame is the frame currently being processed, and the previous frame is the frame one frame before the current frame.

In step S902, the display device control unit 801 ^{obtains the detailed blur amount VnDet} _from the video file. The blur amount is detected by calculating how much the feature point of the image of the current frame has moved from the previous frame.

A known method can be adopted for extracting feature points. For example, a luminance information image is generated by extracting only the luminance information of the image of the frame, and the image obtained by shifting the image by one to several pixels is subtracted from the original image, and the pixels whose absolute value is greater than the threshold value are extracted as feature points. You may Further, an image obtained by applying a high-pass filter to the luminance information image may be subtracted from the original luminance information image, and the extracted edge may be extracted as a feature point.

Differences are calculated a plurality of times while the luminance information images of the current frame and the previous frame are shifted by one to several pixels, and the movement amount is calculated by calculating the position where the difference in the pixel of the feature point becomes small.

Since a plurality of feature points are required as described later, it is preferable to divide the images of the current frame and the previous frame into a plurality of blocks and extract feature points for each block. Although the block division depends on the number of pixels and the aspect ratio of the image, generally 4×3=12 blocks or 9×6=54 blocks is preferable. If the number of blocks is too small, trapezoidal distortion due to tilting of the imaging unit 40 of the camera body 1 and rotational shake in the direction of the optical axis cannot be corrected accurately. This is because the error is included due to the closeness. For this reason, the optimal number of blocks is appropriately selected depending on the number of pixels, the ease of finding characteristic points, the angle of view of the subject, and the like.

To calculate the amount of movement, it is necessary to shift the luminance information images of the current frame and the previous frame by one to several pixels and calculate the difference a plurality of times, resulting in a large amount of calculation. Therefore, since the actual amount of movement is calculated based on the rough blurring amount V _n ^Pre and the deviation (by how many pixels) therefrom, calculating the difference only in the vicinity of the rough blurring amount greatly reduces the amount of calculation. can be reduced.

Next, in step S903, image stabilization processing is performed using the detailed blurring amount _VnDet acquired in step S902, after which this subroutine is ^exited .

Known methods for vibration reduction processing include Euclidean transformation capable of rotation and translation, affine transformation capable of such rotation, and projective transformation capable of trapezoidal correction.

The Euclidean transform can correct movement and rotation in the X-axis direction and Y-axis direction, but cannot correct blur caused by camera shake in the front-rear direction and in the pan/tilt direction that occurs in the imaging unit 40 of the camera body 1 . Therefore, in the present embodiment, image stabilizing processing is performed using affine transformation capable of correcting enlargement, skew, and the like. An affine transformation in which coordinates (x, y) of a reference feature point are moved to coordinates (x', y') is represented by Equation 100 below.

The affine coefficients of the 3×3 matrix of Equation 100 can be calculated if at least three feature point shifts can be detected. However, when the detected feature points are close to each other or are on a straight line, the vibration reduction processing becomes inaccurate at locations farther than the feature points or away from the straight line. Therefore, it is preferable to select feature points that are far apart from each other and do not lie on a straight line. Therefore, when a plurality of feature points are detected, feature points close to each other are omitted and the rest are normalized by the least squares method.

FIG. 18(f) is a diagram showing an image obtained by subjecting the distortion-corrected image shown in FIG. 18(e) to the vibration reduction processing in step S903. Since clipping processing is performed during image stabilizing processing, the image shown in FIG. 18(f) has a smaller angle of view than the image shown in FIG. 18(e).

By performing such anti-vibration processing, it is possible to obtain a high-quality image in which blur is corrected.

A series of operations performed by the camera body 1 and the display device 800 included in the camera system of this embodiment have been described above.

When the user turns on the power switch 11, selects the moving image mode with the imaging mode switch 12, and observes the front without looking up, down, left, or right, first, the face direction detection unit 20 detects the observation direction. Detect vo(vector information [0°, 0°]) (FIG. 12A). After that, the recording direction/angle of view determination unit 30 cuts out the image (FIG. 11B) of the target field of view 125o shown in FIG.

Thereafter, when the user starts observing, for example, the child (subject A131) in FIG. °]) (FIG. 11C). After that, the recording direction/angle of view determination unit 30 cuts out an image with a target field of view of 125 m (FIG. 11C) from the ultra-wide-angle image captured by the imaging unit 40 .

In steps S800 and S900, the display device 800 performs optical correction processing and vibration reduction processing on images cut out in various shapes according to the viewing direction. As a result, even if the general control CPU 101 of the camera body 1 is of low spec, even if a video with a large distortion, for example, a target field of view of 125 m (FIG. 11C) is cut out, a child (subject A 131) is centered as shown in FIG. 11D. An image corrected for distortion and shaking can be obtained. In other words, the user can obtain an image in which his observation direction is captured without touching the camera body 1 except for turning on the power switch 11 and selecting the mode with the imaging mode switch 12 .

The preset mode will now be described. As described above, since the camera body 1 is a small wearable device, the camera body 1 is not provided with operation switches, setting screens, and the like for changing detailed settings. Therefore, in this embodiment, the detailed settings of the camera body 1 are changed on the setting screen (FIG. 13) of the display device 800 as an external device.

For example, consider a case where it is desired to continuously capture the same moving image at a field angle of 90° and a field angle of 45°. Since the angle of view is set to 90° in the normal moving image mode, when performing such image capturing, first, after image capturing in the normal moving image mode, the moving image capturing is finished once, and the display device 800 is moved to the camera body 1. It is necessary to switch to the setting screen and switch the angle of view to 45 degrees. However, during continuous imaging, such operations on the display device 800 are troublesome, and the user may miss the desired image.

On the other hand, if the preset mode is set in advance to a mode for capturing a moving image at an angle of view of 45°, after capturing a moving image at an angle of view of 90°, simply slide the imaging mode switch 12 to "Pre". , can be immediately changed to a zoomed-in moving image with an angle of view of 45°. That is, the user does not need to interrupt the current imaging action and perform the above-described troublesome operations.

The contents to be set in the preset mode include not only the change of the angle of view, but also the stabilization level specified by "strong", "medium", "off", etc., and voice recognition settings that are not explained in this embodiment. may

For example, when the user switches from the moving image mode to the preset mode with the imaging mode switch 12 while continuing to observe the child (subject A131) in the previous imaging situation, the angle of view setting value V is changed from 90° to 45°. be. In this case, the recording direction/angle of view determination unit 30 cuts out the image of the target field of view of 128 m indicated by the dotted line frame shown in FIG.

Even in the preset mode, optical correction processing and image stabilization processing are performed in display device 800 in steps S800 and S900. As a result, even if the overall control CPU 101 of the camera body 1 has a low spec, it is possible to obtain a zoomed-in image centering on the child (subject A131) as shown in FIG. An example in which the angle of view setting value V is changed from 90° to 45° in the moving image mode has been described, but the same applies to the still image mode. In addition, the same is true when the field angle setting value V for moving images is 90° and the field angle setting value V for still images is 45°.

In this manner, the user can obtain a zoomed-in image in which the observation direction of the user is captured only by switching modes with the imaging mode switch 12 on the camera body 1 .

In this embodiment, the case where the face direction detection section 20 and the photographing section 40 are integrally configured in the camera body 1 has been described. It is not limited to this as long as it is worn and the imaging unit 40 is worn on the user's body. For example, the imaging/detecting unit 10 of this embodiment can be installed on the shoulder or on the abdomen. However, if the imaging unit 40 is placed on the right shoulder, the subject on the left side may be blocked by the head. preferable. Further, in the case of the abdomen, a spatial parallax occurs between the imaging unit 40 and the head, so it is desirable to perform observation direction correction calculation for correcting the parallax as described in the third embodiment.

(Example 2)
In a second embodiment, a method for calibrating individual differences and adjustment differences of users wearing the camera body 1 will be described in detail with reference to FIGS. 20 to 23. FIG.

Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of the second embodiment, the same reference numerals are used for the configurations that are the same as those of the camera system of the first embodiment, redundant descriptions are omitted, and details of different configurations are added each time. to explain.

The user who wears the camera body 1 is represented by their physique, the inclination and angle around the neck to which the camera body 1 is attached, the condition of the clothes represented by the collar when worn, and the rest of the adjustment of the band portions 82L and 82R. There are individual differences and adjustment differences. Therefore, the center of the optical axis of the imaging lens 16 of the camera body 1 and the center of the visual field when the user faces the front (hereinafter referred to as the user's natural state) usually do not coincide. For the user, the center of the optical axis of the imaging lens 16 of the camera body 1 is not the center of the recording area (target visual field 125) where the image is to be cut out, but the center of the visual field in the user's posture and movement is the center of the recording area. It is desirable to

In addition to the natural visual field center of the user, there are individual differences in the visual field center when the user turns his head in each direction including up, down, left, and right, and in the movable range of the neck. Therefore, the relationship between the face direction (observation direction) detected by the face direction detection unit 20 and the center position of the target visual field 125 (hereinafter referred to as the visual field center position) set according to the observation direction also varies among individuals. . Therefore, calibration work is required to associate the face direction with the center position of the field of view.

Generally, it is desirable that the calibration operation be performed as part of the preparatory operation process (step S100) in FIG. 7A. Normally, it is assumed that the calibration operation is performed when the camera body 1 is started for the first time. This may be done when positional deviation occurs. The calibration operation may be performed even when the face direction detection unit 20 cannot detect the user's face. Further, when it is detected that the user has attached/detached the camera body 1, the calibration operation may be performed when the user attaches/reinserts the camera body 1 again. In this way, it is desirable that the calibration operation is performed appropriately at the timing determined to be necessary for proper use of the camera body 1 .

FIG. 20 is a diagram illustrating details of the calibrator 850 used for calibration processing according to the second embodiment. In this embodiment, a case where the calibrator 850 also serves as the display device 800 will be described.

Calibrator 850 includes positioning index 851 and calibration button 854 in addition to button A 802, display unit 803, in-camera 805, face sensor 806, and angular velocity sensor 807, which are components of display device 800 shown in FIG. 1D. ing. Note that the button B804 used in the first embodiment is not used in this embodiment and can be replaced with a calibration button 854 as described later, so it is not illustrated here.

20A shows a case where the positioning index 851 is a specific pattern displayed on the display unit 803, and FIG. 20B shows a case where the external appearance of the calibrator 850 is used for the positioning index 851. FIG. In the case of FIG. 20B, a positioning index center 852, which will be described later, is calculated from information on the outer shape of the calibrator 850. In FIG.

The positioning index is not limited to the examples shown in FIGS. 20A and 20B, and may be separate from the calibrator 850, for example. Any positioning index may be used as long as the size is easily measured and the shape is easy for the user to see. For example, it may be a lens cap for the imaging lens 16 or a charging unit for the camera body 1 . In any case, the basic concept of the calibration operation is the same, so the following description will be made mainly using the calibrator 850 shown in FIG. 20(a) as an example.

In this embodiment, the calibrator 850 is assumed to have the function of the display device 800 as well. Also, the calibrator 850 may be a dedicated device, or may be, for example, a general smart phone or a tablet terminal.

The positioning index 851 is an index displayed on the display unit 803 of the calibrator 850, and is a graphic from which the horizontal width L851a of the positioning index, the vertical width L851b of the positioning index, and the positioning index center 852 can be calculated. In the calibration process, which will be described later, the user faces toward the center of the positioning index 851, so it is desirable that the positioning index 851 has a shape that is easy to grasp at the center of the visual field. In FIG. 20(a), a circle with a black dot at the center of the cross is shown, but the shape is not particularly limited to this shape. In addition, it may be a square, triangle, star-shaped figure, or, for example, an illustration of a character.

The positioning index 851 is imaged by the imaging unit 40 of the camera body 1 . Based on the imaged image, the display device control section 801 (position calculating means/distance calculating means) calculates the distance between the photographing/detecting section 10 and the calibrator 850 and the position coordinates of the positioning index 851 appearing on the image range. . In this embodiment, the calibrator 850 having the functions of the display device 800 performs such calculations.

Angular velocity sensor 807 can measure movement of calibrator 850 . Based on the measured value of the angular velocity sensor 807, the display device control unit 801 calculates movement information indicating the position and orientation of the calibrator 850, which will be described later.

The calibration button 854 is a button that is pressed when the user faces the vicinity of the center of the positioning index 851 in calibration processing, which will be described later. In FIG. 20A, the calibration button 854 is a touch button displayed on the touch panel display unit 803, but the button A802 or button B804 may function as the calibration button.

Next, referring to the flow chart of FIG. 21, the calibration process that is executed when a video is extracted from the ultra-wide-angle image captured by the imaging unit 40 according to the direction of the user's face and the video is processed will be described. will be explained in detail.

FIG. 21 is a flow chart of calibration processing according to this embodiment, which is executed in the camera body 1 (first calibration means) and the calibrator 850 .

As an aid to the explanation, in FIG. 21, the step of receiving the user's operation of the camera body 1 and the calibrator 850 is included in a frame in which the user is the subject of the action. In addition, in FIG. 21, the steps executed by the display device control unit 801 of the calibrator 850 in response to the user's operation are included in a frame in which the calibrator 850 is the subject of action. Similarly, in FIG. 21, the steps executed by the overall control CPU 101 of the camera body 1 in response to the user's operation are put in a frame with the camera body 1 as the subject of action.

Specifically, in steps S3104 and S3108 of FIG. 21, the camera body 1 is the subject of action, and in steps S3101, S3105, and S3106, the user is the subject of action. Further, the calibrator 850 is the main operator of steps S3102, S3103, S3106a, S3107, S3107b, and S3110.

When this process starts, first, in step S3101, the user operates the button A802 to turn on the power of the calibrator 850 if the power of the calibrator 850 is not turned on. Similarly, when the camera body 1 is not powered on, the power switch 11 is turned on to turn on the power of the camera body 1 . The user then establishes a connection between the calibrator 850 and the camera body 1 . When this connection is established, the display device control section 801 and the overall control CPU 101 enter calibration mode.

In step S3101, the user wears the camera body 1, adjusts the lengths of the band portions 82L and 82R, the angle of the camera body 1, etc., and arranges the camera body 1 at a suitable position for photographing/detection. The unit 10 is put into a state capable of imaging.

In step S<b>3102 , the display device control unit 801 (first display means) displays the positioning index 851 on the display unit 803 .

Next, in step S3103, the display device control unit 801 instructs the user on the instruction display 855 of the specified position to which the calibrator 850 should be held. In this embodiment, the front, upper right, lower right, upper left, and lower left five points are indicated in order as designated positions. However, the designated position is not limited to this as long as calibration is possible.

In step S3104, the overall control CPU 101 activates the imaging unit 40 to enable imaging, and activates the face direction detection unit 20 to enable detection of the user's face direction.

In step S3105, the user holds the calibrator 850 over the specified position instructed in step S3103.

Next, in step S3106, the user turns the face toward the positioning index 851 while maintaining the position of the calibrator 850 at the specified position, aligns the user's visual field center with the positioning index 851, and presses the calibration button 854. push.

In step S3106a, the display device control unit 801 (second display unit) determines whether or not the user has viewed the positioning index center 852 of the positioning index 851 at the center of the visual field. If it is determined that it has been seen (YES in S3106a), the display device control unit 801 notifies the user to start the calibration of the specified position in the instruction display 855 in step S3107, and redisplays the calibration button 854. If determined as NO in step S3106a, the user repeats the processing from step S3105.

When the user presses the calibration button 854 in step S3107a, the display device control unit 801 transmits a calibration instruction to the camera body 1 in step S3107b.

In step S3108, the overall control CPU 101 (acquisition/detection means) acquires an ultra-wide-angle image in which the positioning index 851 is captured by the imaging unit 40 in response to the calibration instruction from the calibrator 850, and detects the face direction. A face direction is detected in the unit 20 . After that, the overall control CPU 101 (generating means) calculates the position coordinate information of the positioning index center 852 in the super-wide-angle image acquired here, and indicates the relationship between the calculated position coordinate information and the face direction detected here. Generate information.

Details of the processing of steps S3103 to S3108 will be described below with reference to FIGS. 22A to 22F.

22A to 22F are diagrams for explaining the calibration operation in the front direction of the user. By the calibration operation, the center position of the visual field in the user's natural state is matched with the center position of the target visual field 125 in the image captured by the photographing section 40 of the camera body 1 .

FIG. 22A is a diagram showing a screen displayed on the display unit 803 of the calibrator 850 in step S3103 of FIG. 21 during the calibration operation for the front direction of the user.

As shown in FIG. 22A, the display unit 803 of the calibrator 850 displays a positioning index 851 and an instruction display 855 indicating where the user should place the positioning index 851 .

The instruction display 855 is a character string that instructs to arrange the positioning index 851 at the center of the visual field when the face is turned to the front. The instructions displayed as the instruction display 855 are not limited to instructions in characters, and may be instructions in other methods such as illustrations, photographs, and moving images.

Alternatively, the instruction display 855 may be displayed as in a so-called general tutorial, and then the positioning index 851 may be displayed.

FIG. 22B is a diagram showing how the user holds the calibrator forward according to the instructions shown in the instruction display in FIG. 22A.

The user holds the calibrator 850 forward according to the instructions shown in the instruction display 855 in FIG. 22A (step S3105). Then, the user arranges the calibrator 850 so that the positioning index 851 is at the center of the field of view when facing the front of the face, and presses the calibration button 854 (FIG. 22A) (step S3106). Determination in step S3106a is performed in response to pressing of the calibration button 854. FIG. A specific procedure of this determination method will be described later. If YES is determined in step S3106a, the display device control unit 801 changes the instruction display 855 shown in FIG. indicate.

After that, the user presses the calibration button 854 after confirming that the instruction display 855 shown in FIG. 22A has been changed to the notification "Calibration in the front direction is started" (step S3107a).

In response to pressing of the calibration button 854, a calibration instruction is transmitted to the camera body 1 in step S3107b, and the photographing unit 40 acquires a photographed image in step S3108.

FIG. 22C is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens 16 in the state of FIG. 22B, and FIG. 22D is a schematic diagram showing an aberration-corrected image of the ultra-wide-angle image shown in FIG. 22C. .

On the other hand, in step S3108, the face direction detection unit 20 acquires the face direction in response to the user pressing the calibration button 854 in the state of FIG. 22B.

FIG. 22E is a schematic diagram showing the face direction image recorded by the face direction detection unit 20 in step S3108 of FIG. 21 during the calibration operation for the front direction of the user.

As described above with reference to FIGS. 8G to 8K in the first embodiment, the face direction detection unit 20 uses the distances and angles between the chin positions 207, 207r, 207u, etc. and the neck position 206 to determine the left, right, up, and down directions of the face. Calculate the angle of However, the distances and angle values between the chin positions 207, 207r, 207u, etc. and the neck position 206 also have individual differences and adjustment differences, as indicated by the physique of the user, as with the center of the image. inconsistent. Therefore, in this embodiment, the relationship between the chin position and the neck position 206 when the calibration button 854 is pressed is defined as a value when the front is the center of the visual field of the user. This makes it possible to accurately calculate the face direction of the user regardless of individual differences or adjustment differences.

Returning to FIG. 21, in step S3109, the overall control CPU 101 determines whether or not the preparation for calibration in the front direction has been completed. That is, it is determined whether information necessary for calculating the chin position 207, the neck position 206, and the positioning index center 852 has been acquired.

At this time, if the necessary information has not been acquired, it is determined that the preparation for the calibration in the front direction has not been completed (NO in step S3109), and the missing information among the necessary information can be acquired again. , the operation from step S3102 is repeated. It should be noted that when necessary information cannot be obtained, it is of course not necessary to perform all the operations from step S3102, and only necessary operations to obtain the missing information again may be performed again.

Here, the determination in step S3106a is performed using the face sensor 806 or the in-camera 805 mounted on the calibrator 850. FIG. A specific procedure of this determination method will be described below by taking as an example a case where the in-camera 805 is used to perform a calibration operation in the front direction of the user. In the case of using the face sensor 806, although there is a difference whether the information is two-dimensional or three-dimensional, the basic idea is the same, so a description thereof will be omitted. However, when the face sensor 806 is used in the determination in step S3106a, the face direction detector 20 of the camera body 1 emits the infrared ray 23 to the user while the infrared ray 823 is projected from the face sensor 806 to the user. Face detection by projecting light is not performed. This is to prevent the infrared rays 23 and 823 from interfering with each other.

First, when the user presses the calibration button 854 in FIG. 22A in step S3106, the display device control unit 801 captures an image with the in-camera 805 (face detection means), and captures an in-camera image 858 ( FIG. 22F) is obtained. Further, the display device control unit 801 extracts the user's front neck 201, chin 203, and face 204 including the nose from the acquired in-camera image 858, and the position information of the image capturing/detecting unit 10 (image capturing unit 40). to detect

In step S3106a, the display device control unit 801 (determining means) determines whether the user is looking at the positioning index center 852 of the positioning index 851 in the center of the visual field using each position information detected by the in-camera image 858. do.

As a result of this determination, when it is determined that the user is looking in a different direction, the display device control section 801 displays information to the effect that correct information cannot be obtained on the instruction display 855 . As a result, the user can be instructed to redo the calibration operation.

Note that the display device control unit 801 uses the in-camera image 858 to determine whether the correct calibration operation is performed when the shooting/detection unit 10 is tilted more than a certain amount, or when the face direction detection window 13 is blocked or dirty. In some cases, it can be determined that it is not possible. In such a case as well, the display device control unit 801 may display information to the effect that correct information cannot be obtained on the instruction display 855 .

Furthermore, using the in-camera image 858 acquired in step S3106a and the ultra-wide-angle image acquired in step S3108, it is possible to acquire information necessary for parallax correction, which will be described later in the fifth embodiment.

Specifically, before the positioning index 851 is imaged by the imaging unit 40 in step S3108, information on the size of the positioning index 851 (horizontal width L851a and vertical width L851b) is transmitted from the calibrator 850 to the camera body 1 in advance. back. As a result, the overall control CPU 101 uses information on the size of the positioning index 851 and the image of the positioning index 851 appearing in the ultra-wide-angle image acquired in step S3108 to determine the distance between the photographing/detecting unit 10 and the positioning index 851. can be calculated. The positioning index 851 is located in the calibrator 850, which is the same housing as the in-camera 805. In FIG. 22B, the calibrator 850 faces the user substantially.・Equal to the distance between the detection unit 10 and the positioning index 851 .

Similarly, before the in-camera image of FIG. 22F is captured by the in-camera 805 in step S3106a, information about the size of the imaging/detecting unit 10 is transmitted from the camera body 1 to the calibrator 850 in advance. As a result, the display device control unit 801 (vertical distance calculation means) uses the size information of the photographing/detecting unit 10 and the image of the photographing/detecting unit 10 captured in the in-camera image 858 in FIG. A vertical distance 5070 between the center of the 16 optical axes and the user's viewpoint position can be estimated. In addition, the display device control unit 801 can also estimate the distance 2071 between the imaging lens 16 and the chin 203 of the user. The distance 2071 may be the distance between the face direction detection window 13 and the tip of the chin 203 .

Here, in order for the face direction detection unit 20 to calculate the user's neck position 206 and the chin/BR> position, the user's face must be positioned within the face direction detection window according to the design of the face direction detection unit 20 . 13 must be at least a certain distance away. Therefore, this estimation result can be used as one of the determination conditions when determining whether the face direction detection unit 20 can correctly detect the face direction.

Returning to FIG. 21, if the overall control CPU 101 determines in step S3109 that the necessary information has been acquired and the front direction calibration preparation has been completed by the method described above, the process advances to step S3110.

In step S3110, the display device control unit 801 (first calibration means) calculates information necessary for offsetting the cutout center position so as to absorb individual differences and adjustment differences, and cuts out based on the information. Offset the center position.

A specific description of the calculation in step S3110 is as follows.

If the user is in an ideal state according to the design values and is ideally wearing the camera body 1, then the center 856 of the ultra-wide-angle image acquired in step S3108 shown in FIG. The position of the positioning index center 852 to be photographed should approximately match. However, in reality, the positions of the center 856 and the positioning index center 852 in the ultra-wide-angle image usually do not match because of individual differences and adjustment differences such as the physique of the user described above.

For the user, the cutout center position is not the center 856 of the ultra-wide-angle image shown by the camera body 1, but the field-of-view center of the user's posture and movement, that is, the position of the positioning index center 852 in the ultra-wide-angle image. .

Therefore, the amount of deviation between the positioning index centers 852 and 856 in the ultra-wide-angle image is measured, and the cutout center position is offset to a position based on the positioning index center 852 instead of the center 856 of the ultra-wide-angle image. Also, the face direction detected by the face direction detection unit 20 at that time is also offset by the same method.

A specific offset method will be described with reference to FIGS. 22C and 22D. As shown in FIG. 22C, the displacement amount of the positioning index center 852 with respect to the center 856 of the ultra-wide-angle image is measured, and this is divided into a horizontal displacement amount 857a and a vertical displacement amount 857b. The offset amount may be determined after the conversion process is performed.

Alternatively, as shown in FIG. 22D, the offset amount may be determined after the super-wide-angle image is subjected to appropriate conversion processing according to the projection method. That is, the amount of deviation between the center 856a and the center 852 of the positioning index in the super-wide-angle image after conversion is measured, and the amount of deviation is divided into the amount of deviation 857c in the horizontal direction and the amount of deviation 857d in the vertical direction to determine the offset amount. good.

Which of the offset methods shown in FIGS. 22C and 22D should be used can be arbitrarily determined in consideration of the processing load and purpose of the camera system.

By performing the calibration operation in the front direction described above, regardless of individual differences and adjustment differences, each user's face direction at the time of wearing, the visual field center in that face direction in the ultra-wide-angle image, and the face It is possible to appropriately associate the face direction of the direction detection unit 20 .

Up to this point, the calibration operation in the front direction out of the five directions of the front, upper right, lower right, upper left, and lower left has been explained. It is necessary to perform also for four directions.

Therefore, in FIG. 21, when the process of step S3110 is complete|finished, it progresses to step S3111.

In step S3111, if it is determined that there is a direction in which the calibration operation has not yet been performed among the five directions of the front, upper right, lower right, upper left, and lower left, the direction in which the calibration operation is performed is changed to that one direction. and returns to step S3103. As a result, the calibration operation is repeated in the same manner for the remaining directions other than the already completed front direction.

Although not shown in FIG. 21, if it is determined in step S3111 that there is no direction for which the calibration operation has not been performed, this processing ends.

23A to 23E are diagrams for explaining the calibration operation for the upper right direction of the user (upper right direction in the ultra-wide-angle image). 23A to 23E correspond to FIGS. 22A to 22E, respectively, and the basic operation is the same, so common explanations are omitted.

Here, as shown in FIG. 23A, an instruction display 855 displays a text instruction to place the positioning index 851 in the center of the visual field when the face is turned to the upper right.

FIG. 23B is a diagram showing how the user holds the calibrator 850 to the upper right according to the instruction indicated by the instruction display 855 in FIG. 23A.

FIG. 23C is a schematic diagram showing the entire super-wide-angle image captured by the imaging lens 16 in the state of FIG. 23B.

As shown in FIG. 23C, as a specific offset method, first, the amount of positional deviation between the center 856 and the positioning index center 852 in the ultra-wide-angle image is measured. After that, the measured deviation amount is divided into a diametrical deviation amount 857e and an angular deviation amount 857f, and an offset amount is determined after appropriate conversion processing is performed according to the projection method for all angles of view.

Alternatively, as shown in FIG. 23D, the offset amount may be determined after performing appropriate conversion processing on the ultra-wide-angle image according to the projection method. That is, the amount of deviation between the center 856a and the center 852 of the positioning index in the converted ultra-wide-angle image may be measured, and the amount of deviation may be divided into the amount of deviation 857g in the diameter direction and the amount 857h of deviation in the angular direction to determine the offset amount. .

Incidentally, in determining the offset amount described with reference to FIGS. 22A to 22E, a method of dividing the amount of deviation into the vertical direction and the horizontal direction was used. On the other hand, in the determination of the offset amount explained using FIGS. 23A to 23D, the method of dividing the amount of deviation into the diametrical direction and the angular direction was used, but the difference in this method is only for convenience of explanation. Either method may be used.

At this time, as shown in FIG. 23E, the face direction detection unit 20 can acquire the neck position 206 and the chin position 207ru necessary for calculating the face direction when the user faces the upper right direction. there is Therefore, it is possible to correctly measure the face direction when the user looks at the direction of the positioning index center 852 (the upper right direction in this case) regardless of individual differences and adjustment differences of the user.

As described above, in the calibration process shown in FIG. 21, the calibration operation is performed not only in the front direction but also in the upper right, lower right, upper left, and lower left directions. As a result, when the user shakes his or her head in any of the vertical and horizontal directions, the face direction detection unit 20 can accurately measure the direction in which the user is facing, and this allows for individual difference and adjustment. The camera body 1 can be used appropriately regardless of the difference.

In the above description, for the sake of simplification, the method of repeatedly performing the calibration operation in the five directions of the front, upper right, lower right, upper left, and lower left has been explicitly described.

However, the calibration operation is not limited to this method. For example, the user continuously moves the calibrator 850 along a Z-shaped, spiral, polygonal, or other trajectory according to the instruction display 855, and at the same time captures the positioning index 851 being displayed on the calibrator 850 at the center of the visual field. You can find a way to continue. According to this method, the display device control unit 801 transmits calibration instructions to the camera body 1 multiple times while the calibrator 850 is moving according to this method. The overall control CPU 101 acquires the face direction detected by the face direction detection unit 20 each time a calibration instruction is received, and the position coordinate information of the positioning index center 852 in the ultra-wide-angle image captured by the image capturing unit 40, and obtains the history information. hold as Thereafter, the overall control CPU 101 combines the information extracted from the acquired history information to calculate the relationship between the clipping center position of the image and the face direction of the user. Furthermore, using the information of the in-camera 805 and the face sensor 806 acquired by the calibrator 850 while the calibrator 850 is moving by this method, the information extracted from the history information can be read by the user by looking at the positioning index 851. You may limit to the information of the state in which it exists. As a result, for example, information about a state in which the user is looking away is not extracted from the history information, so it is possible to improve the accuracy of calculating the relationship.

Further, the display device control unit 801 may also transmit the measurement value of the angular velocity sensor 807 to the camera body 1 when instructing calibration. In this case, the overall control CPU 101 acquires movement information indicating the movement method of the calibrator 850 by the user and the position and orientation of the calibrator 850 from the transmitted measured values by the angular velocity sensor 807, and also retains this as history information. . As a result, the calibration operation is performed based on the movement information based on the measured value by the angular velocity sensor 807, the face direction detected by the face direction detection unit 20, and the position coordinate information of the positioning index center 852 in the ultra-wide-angle image captured by the image capturing unit 40. It can be done easily and accurately.

However, in this case, the movement information based on the measured value by the angular velocity sensor 807 and the movement information based on the position coordinate information of the positioning index 851 must match. Therefore, when using the measured value by the angular velocity sensor 807, it is necessary to synchronize communication between the camera body 1 and the calibrator 850. FIG.

As described above, in the second embodiment, the calibration method for relating the direction of the face of the user and the center position of the target field of view 125 in the ultra-wide-angle image even if there are individual differences and adjustment differences has been described. However, the present invention is not limited to the various forms exemplified in Example 2, and various modifications and changes are possible within the scope of the gist of the present invention.

(Example 3)
In the third embodiment, a method for preventing motion sickness due to secondary recorded video will be described with reference to FIGS.

Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of the third embodiment, the same reference numerals are used for the configurations that are the same as those of the camera system of the first embodiment, and redundant descriptions are omitted, and details of different configurations are added each time. to explain.

Advances in imaging technology have made it possible to easily enjoy CG and powerful 3D images that look like live action.

On the other hand, when such 3D images are images with dynamic movements such as VR images or images with a lot of camera shake, motion sickness is likely to occur when viewing these images. Motion sickness can also cause motion sickness-like symptoms, and there is growing interest in its safety measures.

It is assumed that the camera system is designed to extract and develop the image in the direction in which the user's face is facing in the recording area development process (step S500). In this case, when the user's face moves quickly when the image is captured by the image capturing unit 40 (step S400), the video scene also switches at a high speed.

The user who quickly moved his/her face when the image was captured by the imaging unit 40 does not get motion sickness. can cause

Japanese Patent Application Laid-Open Nos. 2003-100001 and 2003-200010, which record the direction in which the face is facing, do not include any description of countermeasures against motion sickness.

Therefore, in the present embodiment, even if the user moves his/her face quickly when the photographing unit 40 takes an image, control is performed so that the resulting image does not include image scenes that switch at a high speed, thereby preventing the viewer from motion sickness. To provide a camera system that prevents

As described with reference to FIGS. 8H to 8K and FIGS. 10B to 10D, the user's face rotates vertically and horizontally when the photographing unit 40 takes an image.

Therefore, hereinafter, the direction and speed of movement of the user's face are represented by angular velocity ω, and the amount of movement is represented by angle Θ.

The angular velocity ω is calculated by dividing the angle Θ detected by the face direction detector 20 by the detection interval.

Here, examples of actions in which a person quickly moves his/her face include looking back, glimpsing, and observing a moving object.

Looking back is an action of vigorously turning back, for example, when a loud sound is generated behind the user.

A glimpse is an action of returning to the original position of the face because there was a change in the field of vision that caused a concern, and the person saw it but was not interested in it.

Observing a moving object is, for example, an action of observing a bird or a kite flying freely in the sky.

When these operations occur when the photographing unit 40 takes an image, if the image in the direction in which the user's face is facing is cut out and developed by the recording area development processing, as described above, the viewer of the completed image will not get motion sickness. can cause

Therefore, when the angular velocity ω equal to or greater than the threshold value ω0 is calculated for a predetermined time or longer (a first predetermined time or longer), the overall control CPU 101 determines whether the user makes a quick facial movement (looks back, glances, or observes a moving object). ) has occurred. Furthermore, when the overall control CPU 101 determines that the action that has occurred is not a glance or a moving object observation by the method described later with reference to FIG. 25, it determines that the action is looking back. In this case, the overall control CPU 101 does not cut out the image in the direction in which the user's face is facing as it is by the recording area development processing, but performs delayed cutout in which the image is cut out with a delay with respect to the movement of the user's face.

Here, in this embodiment, the threshold ω0 is set to π/8 rad/s. This is the speed at which the face moves from 0 degrees in front to 90 degrees to the side in 4 seconds. However, the threshold ω0 is not limited to π/8 rad/s. For example, based on the frame rate n fps, the threshold ω0 may be set to (n×π)/x rad/s (where x is an arbitrary value).

The angular velocity ω is the angle Θ _n acquired from the image of the current frame n and its acquisition time t _n , and the angle Θ _{n−1 acquired from the image of the previous frame n−1} and its acquisition time t _n−1 . , the angular velocity _ωn can be obtained by the following equation.
ω _n =(Θ _n -Θ _n-1 )/(t _n -t _n-1 )
However, the angular velocity ω may be an addition average of x frames from the angular velocity ω _n− x of the frame n−x before x times to the angular velocity ω _n of the current frame n.

Furthermore, in this embodiment, the predetermined time is set to 0.2 seconds, but it is not limited to this value.

Delayed clipping when the user is looking back will be described below with reference to FIG.

The description of FIG. 11 and FIG. 12 in Example 1 considered the distortion of the imaging lens 16, but in this example, the distortion of the imaging lens 16 is not considered for the sake of simplicity of description. Further, it is assumed that the calibration process of the second embodiment is performed on the frame images, and the center of each frame image coincides with the user's visual field center when the image is captured. In addition, in order to explain the case where the face is turned to the side, the case where light rays up to the maximum FOV angle of 192° are projected onto the solid-state imaging device 42 will be explained as an example.

A region 4000 represents the imageable pixel region of the solid-state imaging device 42 .

An image 4001 (FIGS. 24(a) and 24(b)) is an image of a frame _fn cut out as the target field of view 125 in the direction the face is currently facing.

An image 4002 (FIGS. 24(a) and 24(b)) is an image of the frame f _n−1 extracted as the target field of view 125 in the direction the face was facing last time.

Hereinafter, let d be the value of the distance 4010 (FIG. 24(a)) from the center of the image 4002 of the frame f _n−1 to the center of the image 4001 of the frame f _n .

An image 4003 (FIG. 24(b)) is projected onto the area 4000 as an image of the delayed clipped frame f' _n when the angular velocity ω of the face based on the face direction detected by the face direction detection unit 20 is equal to or greater than the threshold value ω0. It is an image cut out from the video to be processed.

Hereinafter, let d' be the value of the distance 4011 from the center of the image 4002 of the frame fn _-1 to the center of the image 4003 of the delayed clipped frame _f'n .

The delay distance 4012 is the distance from the center of the image 4001 of frame fn to the center of the image 4003 of frame _f'n , and its value is hereinafter _d ''.

At this time, the value d of the distance 4010 is greater than the value d' of the distance 4011 (d>d').

Next, an example of how to determine the value of d' will be described with reference to FIG. 24(c).

Here, a case will be described where the user quickly moves the face from the front (viewing direction vo (vector information [0°, 0°])) to the right by 90 degrees. In this case, after an image ₄₀₂₁ of frame fn in which the face faces the front (observation direction vo (vector information [0°, 0°])) is obtained, after a short period of time, the face is 90° to the right. An image 4022 of the facing frame f _n+x is obtained.

In order to prevent motion sickness, if it is necessary to move from the front to the right side to 90 ° over t seconds (eg, 4 seconds) or more, if the video frame rate is n fps (eg, 30 fps), d '=(f _n+x −f _n )/(n fps×t seconds).

On the other hand, as the distance _d '' from the frame fn to the frame _f'n increases, the subject that the user was looking at is not reflected in the frame _fn because the direction in which the user's face is facing is not set as the recording direction. There is a possibility that there is no.

Therefore, when the delay time becomes equal to or longer than a predetermined time _Thdelay (second predetermined time), the delayed clipping is stopped, and the clipping is performed in the direction in which the face is currently facing.

Here, the delay time is the difference between the time t0 (step S4211 in FIG. 26) at which the delay begins to occur and the current time tn (step _S4213 in FIG. 26) at which the face continues to move.

Although the predetermined value Th _delay is set to 1 second in this embodiment, it is not limited to this. For example, based on the frame rate n fps, the predetermined value Th _delay may be set to 20/n seconds. When the predetermined value Th _delay is 20/n seconds, the higher the frame rate, the shorter the predetermined value Th _delay . This is because the higher the frame rate, the lower the possibility of motion sickness, so it is possible to return to the cutout process in the direction in which the face is currently facing with a short delay time.

On the other hand, if the delayed clipping is canceled and the clipping is resumed in the direction in which the face is currently facing, the video scene will suddenly switch. Since such abrupt switching of video scenes is unnatural for users, fade-outs, fade-ins, or other video effects may be incorporated.

Also, the trajectory of the direction in which the face is currently facing is held so that the extraction of the direction in which the face is currently facing can be restarted.

Here, a case where it is determined that the user is glancing at the trajectory of the direction in which the held face faces will be described with reference to FIG. 25(a).

The process of canceling delayed clipping and returning to clipping in the direction in which the face is currently facing is executed even when the delay time exceeds the predetermined value Th _delay , as described above. is also executed if it turns back immediately after looking in a particular direction.

FIG. 25(a) is a diagram showing an example of the trajectory of the direction in which the user's face is facing while glancing.

Position 4101, the center of the image in frame f _n−3 , coincides with the user's visual field center when the face begins to move. The user's visual field center then moves to positions 4102, 4103, 4104, which are the image centers of frames f _n-2 , f _n-1 , f _n , respectively. Hereinafter, such a motion of the user's visual field center is referred to as a face motion vector.

The user's visual field center then stays for a while at position 4104 before moving to positions 4105, 4106, and 4107, which are the centers of the images of frames fnx ₊₁ , fnx ₊₂ , and fnx+ ₃ , respectively, and of the image of frame fnx ₊₃ . Stop at position 4107, which is the center.

That is, the face motion vector at positions 4101 to 4104 and the face motion vector at positions 4104 to 4107 are in opposite directions.

When the general control CPU 101 detects a frame group in which the face motion vectors match in opposite directions, as illustrated in FIG.

In this case, the overall control CPU 101 performs delayed extraction from a position 4101 where the face starts to move to a position 4104 where the motion vector of the face starts to move in the opposite direction. This is because the position 4104 is considered to be the position where the subject that the user wants to glance at appears.

On the other hand, after performing the delayed clipping up to the position 4104, the overall control CPU 101 resumes clipping in the direction the face is currently facing up to the position 4107 where the movement of the face has stopped.

Furthermore, when the user's face is moving, the overall control CPU 101 detects an object existing near the center of the visual field in the direction the face is facing, and detects the detected object in the direction the face is facing. If the user stays in the center of the field of view, it is determined that the user is observing a moving object. In this case, the present embodiment does not perform delayed clipping.

FIG. 25(b) is a diagram showing an example of an image of each frame when the user is observing a moving object.

The position, which is the center of the image 4121 of frame f _n−1 , coincides with the user's visual field center when the face starts to move. The user's visual field center then moves to a position that is the center of each of the images 4122-4126 of frames f _n , f _n+1 , f _n+2 , f _n+3 , and f _n+4 .

A bird, which is the same subject, continues to exist near the center of images 4121 to 4126 of each frame.

When the overall control CPU 101 detects a continuous frame group in which the same subject exists near the center of the image, as illustrated in FIG. 25B, the CPU 101 determines the frame group to be a moving object observation frame group.

In this case, the overall control CPU 101 does not perform delayed clipping. This is because there is a high possibility that the subject will not appear in the video if delayed clipping is performed when observing a moving object.

In addition, if the viewer watches a video in which the images 4121 to 4126 are cut out according to the fast facial movement of the user when observing a moving object, it may cause motion sickness. Therefore, the overall control CPU 101 does not cut out images for the frame group during observation of the moving object, and records the entire pixel area imageable by the solid-state imaging device 42 , ie, the image of the area 4000 .

Note that the threshold value ω0, the predetermined time period, and the predetermined value _{Th_delay} described above may have a width called a dead zone.

Next, motion sickness prevention processing according to this embodiment will be described with reference to the flowchart of FIG. 26 . It should be noted that this processing is executed each time frame shooting in step S400 is performed while the shooting unit 40 is shooting a moving image.

In step S4201, the overall control CPU 101 acquires the face direction (observation direction) recorded in the primary memory 103 in the face direction detection processing executed for the current frame shooting.

In step S4202, the overall control CPU 101 acquires the position and size (cutout range) of the video recording frame recorded in the primary memory 103 in the recording direction/range determination processing executed for the current frame shooting.

In step S4203, the overall control CPU 101 (computing means), based on the face direction at the time of the current frame shooting acquired at step S4201, the face direction at the time of the previous frame shooting held in the primary memory 103, and the frame rate, Calculate the angular velocity ω of the face. Thereafter, the overall control CPU 101 determines whether or not the face has started to move at an angular velocity ω equal to or greater than the threshold ω0. Specifically, when the user's face moves at an angular velocity ω equal to or greater than the threshold ω0 for a predetermined time (0.2 seconds) or longer, it is determined that the face has started to move at an angular velocity ω equal to or greater than the threshold ω0. If it is determined that the movement has started, the process advances to step S4204; otherwise, the process returns to step S4201. That is, even if the user's face moves at an angular velocity ω equal to or greater than the threshold ω0, if the time is less than the predetermined time (less than the first predetermined time), the process returns to step S4201. Also, if the primary memory 103 does not hold the face direction at the time of the previous frame shooting and the angular velocity of the face cannot be calculated in step S4203, the process returns to step S4201.

In step S4204, the overall control CPU 101 determines whether or not the face has moved by a predetermined angle or more based on the angular velocity ω of the face calculated in step S4203. If determined to have moved, the process advances to step S4206; otherwise, to step S4205. In step S4204, the overall control CPU 101 may determine whether or not the face has moved at a predetermined angular velocity or more for a predetermined period of time (0.2 seconds) or longer.

In step S4205, the overall control CPU 101 determines whether or not the movement of the face has stopped based on the angular velocity ω of the face calculated in step S4203. If it is determined that it has stopped, the process returns to step S4201, and if it is determined otherwise, the process returns to step S4204.

In step S4206, the overall control CPU 101 determines whether or not the subject being imaged is moving (whether or not the user is observing a moving object). If it is determined that the object is moving, the process proceeds to step S4207; otherwise, the process proceeds to step S4208.

In step S4207, the overall control CPU 101 determines that cropping development processing will not be performed in the recording area development processing of the current frame, and development processing will be performed on the entire area RAW data acquired from the entire solid-state imaging device 42. Proceed to S4205.

In step S4208, the overall control CPU 101 stores in the primary memory 103 the face direction obtained in step S4201 at the time of frame shooting this time, and proceeds to step S4209.

In step S4209, the overall control CPU 101 (delaying means) performs crop development processing on the cutout range centered at a position shifted by distance d from the face direction of the previous frame in the recording range development processing of the current frame (delayed processing). cut out). After that, the process proceeds to step S4210.

In step S4210, overall control CPU 101 determines whether or not delay time start time t0 stored in primary memory 103 has been cleared. If it is determined to be cleared, the process proceeds to step S4211; otherwise, to step S4212.

In step S4211, overall control CPU 101 stores the current time in primary memory 103 as start time t0, and proceeds to step S4212.

In step S4212, the overall control CPU 101 determines whether or not the movement of the face has stopped before the delay time reaches a predetermined value Th _delay , based on the angular velocity ω of the face calculated in step S4203. If determined to have stopped, the process advances to step S4215; otherwise, to step S4213.

In step S4213, overall control CPU 101 stores the current time in primary memory 103 as time tn, and proceeds to step S4214.

In step S4214, the overall control CPU 101 calculates the delay time, which is the difference between the time tn stored in the primary memory 103 and the start time t0, and determines whether or not the delay time is equal to or greater than a predetermined time _Thdelay . If it is equal to or longer than the predetermined time Th _delay , the process proceeds to step S4215; otherwise, the process returns to step S4206.

In step S4215, overall control CPU 101 clears start time t0 stored in primary memory 103, and proceeds to step S4216.

In step S4216, the overall control CPU 101 determines the recording direction and the angle of view by the recording direction/angle of view determination unit 30 based on the face direction detected by the face direction detection unit 20, and proceeds to step S4217.

In step S4217, overall control CPU 101 records a flag in the metadata of the current frame, and returns to step S4201. The metadata flag attached here is used as the timing for adding video effects (fade effects) such as the fade-in/fade-out described above at the time of secondary recording described in step S1000 of the first embodiment.

As described above, in this embodiment, when the angular velocity ω of the face is equal to or greater than the threshold ω0, the frame in the direction in which the face is facing is not clipped as it is, but is clipped according to the movement of the face, thereby reducing motion sickness. There is

(Example 4)
In the fourth embodiment, a method of correcting the clipping range of the image according to the moving speed of the user's face will be described with reference to FIGS. 27 and 28. FIG.

Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of the fourth embodiment, the same reference numerals are used for the configurations that are the same as the configuration of the camera system of the first embodiment, redundant explanations are omitted, and details of different configurations are added each time. to explain.

First, the operation of changing the observation direction by a person will be described. Normally, when a person finds an object of interest at the edge of the field of vision that is off the center of the field of vision and directs the direction of observation there, the face moves first, and after a certain amount, the body follows with a delay.

That is, in such a case, the direction of the imaging lens 16 on the imaging/detecting unit 10 (FIG. 10(a)) located in front of the clavicle does not move when only the face changes its orientation at the initial movement. After that, when the user turns his/her whole body, the direction of the imaging lens 16 of the camera body 1 also moves. Based on the fact that there are such characteristics of human behavior, the following description will be given.

Further, when the face direction detection unit 20 detects the face direction, variations occur due to detection errors. If the clipping position of the video is calculated based on the detection result of the face direction including such variation, the moving image secondary-recorded in step S1000 will have a blur similar to general moving image shake, and the appearance will be poor. Therefore, in order to correct the shake of fine detection, a low-pass filter is applied to the detection result of the face direction to remove fine variations.

Also, if the direction of the face is detected by following a momentary movement of the face, such as when checking left and right while walking on a public road, the moving image secondary-recorded in step S1000 will easily cause motion sickness. Therefore, in the present embodiment, a process of removing (smoothing) the slight movement component in the direction of the face detected following a momentary movement of the face for about 1 or 2 seconds is performed. As a result, the moving image secondary-recorded in step S1000 can be made to look good.

Next, an outline of extraction range correction processing in this embodiment will be described with reference to FIG. 27 .

The horizontal axis of each graph shown in FIG. 27 represents time, and the vertical axis represents the actual observation center (FIG. 27(a)), the face direction (FIGS. 27(b) and (c)), and the direction of the imaging lens 16, respectively. (FIG. 27(d)) and the angle of the cutout position (FIGS. 27(e) and (f)). Note that the upward direction of the vertical axis indicates the right direction.

FIG. 27A is a graph showing the actual movement of the observation center (face direction). The angle on the vertical axis in FIG. 27(a) does not represent the face direction detected by the face direction detection unit 20, but represents the user's face position from a fixed position such as the ground (ground reference). there is That is, the graph of FIG. 27(a) shows that the user is facing forward at first, but starts to turn to the right from around 1 second.

FIG. 27(b) is a graph showing the detection result (observation direction vi) of the face direction detection unit 20. As shown in FIG. The reason why the line indicating the detection result in FIG. 27(b) is not smooth is that the detection result varies due to the detection error, as described above. Therefore, in this embodiment, the detection result of the face direction detection unit 20 is subjected to a low-pass filter.

In addition, although not detected in FIG. 27B, a process of removing (smoothing) changes in the direction of the face detected following a momentary movement of the face is also performed.

FIG. 27(c) is a graph showing a case where the detection result of the face direction detection unit 20 in FIG. 27(b) is smoothed by applying a low-pass filter. As shown in FIG. 27(c), the line indicating the detection result in FIG. 27(b) becomes a smooth line by applying a low-pass filter. However, by applying such a filter, in FIG. 27(c), the movement of the detected face direction from the front to the right starts from around 2 seconds, and the relevant movement occurs almost at the same time as in the case of FIG. 27(a). There is a delay (time lag) from the beginning in FIG. 27(b). 27(b) and 27(c) are angles from the direction of the imaging lens 16 (camera body 1 reference), unlike the ground reference in FIG. 27(a).

In addition, in FIG. 27(b), the slope becomes gentler from around 4 seconds as compared with FIG. 27(a). As shown in FIG. 27(d), the camera body 1 (in the direction of the image pickup lens 16) begins to move together with the user's body from around 4 seconds, so the face direction detection unit 20 relatively detects this. This means that the movement speed in the face direction is decelerating.

As shown in FIG. 27(e), the cutout position is obtained by adding the movement amount of the camera body (FIG. 27(d)) to the face direction detection result (FIG. 27(c)) smoothed with a low-pass filter. That is, a method of calculating the observation direction that is the center of the target visual field 125 is conceivable. However, when the cropping position is calculated by this simple addition method, the cropping position does not follow the actual movement of the observation center, and the image created by the secondary recording is around 4.5 seconds when the body movement starts. The image appears as if the panning suddenly accelerated from the beginning.

That is, in order to eliminate the sense of incongruity caused by the actual movement of the observation center, it is preferable to calculate the cutout position (expected value) so that the panning is substantially constant as shown in FIG. 27(f).

Therefore, in this embodiment, the cut-out position is calculated so that the panning does not appear to have suddenly accelerated as shown in FIG. 27(e). As shown in FIG. 27, when there are two kinds of moving speeds of the clipping position, 0°/second and 10°/second, the face direction detection result shown in FIG. 27(d), the expected value of FIG. 27(f) can be calculated. However, in reality, there are only two types of movement speeds for the cutout position, that is, the observation direction does not suddenly accelerate or suddenly stop, but slowly decelerates. A slow deceleration curve cannot be drawn. Therefore, in this embodiment, when the movement of the camera body 1 stops, the expected value is decelerated by allocating the moving speed of the clipping position from the start of the movement of the observation direction until it stops or from the past fixed period to several frames. Draw a curve.

Hereinafter, the extraction range compensation/BR>c processing in this embodiment will be described in order using the flowchart of FIG.

In the following, the description of the parts common to the above-described first to third embodiments will be simplified or omitted.

FIG. 28A is a flowchart of a subroutine of the recording direction/range determination process in step S300 of FIG. 7A according to this embodiment.

In step S4000a, smoothing is performed by applying a low-pass filter to the viewing direction vi acquired in the face direction detection processing in step S200 (smoothing means). This is because, as described above with reference to FIG. 27(b), the viewing direction vi varies slightly due to detection errors. For the low-pass filter, it is convenient to take a simple moving average of the past several times, for example, 5 to 10 times. However, at this time, as the number of times of averaging increases, the follow-up when the face direction moves is delayed. In addition, when the user immediately turns to the left after turning to the right, there is also a problem that the viewing direction vi when the user turns to the right most cannot be detected.

Furthermore, since the degree of detection error mixture varies depending on the detection method, it is also preferable to appropriately change the degree of smoothing according to the detection method. It is also possible to apply a low-pass filter differently in the vertical direction and in the horizontal direction.

Also, as described above, there are many cases in which it is not necessary to record a momentary facial movement in order to record what the user has experienced as a video. For example, the face is reluctantly moved to confirm left and right safety while walking as described above. Images captured at such moments need not be recorded. Therefore, in this embodiment, the observation direction vi acquired when the direction returns to the original direction in about 2 seconds is also smoothed in step S4000a.

Further, it is often necessary to confirm safety in the horizontal direction and downward direction, but there is little need to confirm safety in the upward direction. Therefore, the low-pass filter may not be applied in the upward direction.

After the extraction range is determined by the processing of steps S301 to S304 (FIG. 7D), the process advances to step S4000, and the overall control CPU 101 (second calibration means) executes extraction range correction processing.

Thereafter, after recording the corrected cutout range in step S305, this subroutine is exited. The clipping range correction processing will be described with reference to the flowchart of FIG. 28(b).

FIG. 28(b) is a flow chart of the extraction range correction processing in step S4000.

In FIG. 28B, first, in step S4001, the overall control CPU 101 (movement speed calculation means) acquires gyro information from the angular velocity sensor 107, that is, the movement of the camera body 1 in the current frame (gyro movement amount).

Although the angular velocity sensor 107 is used in this embodiment, it is not limited to this as long as the movement of the camera body 1 can be detected. For example, a magnetic sensor (not shown) that measures the magnitude and direction of a magnetic field (magnetic field) may be used, or an acceleration sensor 108 that detects acceleration may be used. Furthermore, a method of detecting a movement vector by extracting a feature point and calculating how much the feature point has moved may be used to calculate the amount of movement of the camera body 1 . A known method can be adopted for extracting feature points. One example is to apply a band-pass filter to an image obtained by extracting only the luminance information of two images to extract the edge, subtract the multiple edge images in a shifted state, and calculate the position where the difference is small. can be used to calculate the amount of movement. Although this method increases the amount of calculation, it is one of the preferred modes because hardware such as the angular velocity sensor 107 is not required, and the weight of the camera body 1 can be reduced.

In the following, the case where gyro information is acquired from the angular velocity sensor 107 will be described as an example.

In step S4002, the movement speed of the camera body 1 (gyro movement speed) is calculated from the gyro information acquired in step S4001 and the gyro information acquired in the past.

In step S4003, it is determined whether the gyro movement speed calculated in step S4002 is decelerating. If the moving speed has not decelerated (NO in step S4003), the process proceeds to step S4004; otherwise, the process proceeds to step S4006.

In step S4004, the overall control CPU 101 (second calibration means/observation direction correction means) calculates the movement speed of the cutout position from the cutout position determined in step S304 and the cutout positions obtained in the past. Next, the overall control CPU 101 obtains a subtraction amount obtained by subtracting the gyro movement speed obtained earlier by the time lag caused by applying the low-pass filter from the calculated movement speed of the clipping position.

In step S4005, the overall control CPU 101 stores in the primary memory 103 the moving speed of the clipping position and the amount of subtraction obtained in step S4004, and exits this subroutine.

In step S4006, the overall control CPU 101 calculates the sum of the subtraction amounts stored in the primary memory 103 so that the change in the movement speed of each clipping position also stored in the primary memory 103 in the past fixed period is constant. Allocate, calculate the expected value, and exit this subroutine. The fixed past period may be a period from when the cutout position actually started to move to the present, or a period from when the angular velocity sensor 107 detects the movement of the camera body 1 to the present. Also, in order to simplify the process, the period may be fixed at about 0.5 seconds to 3 seconds. Note that the expected value before the certain past period is set to the movement speed of the clipping position acquired in step S4004.

Table 1, shown below, shows the displacement (velocity) of the data graphed in FIGS. 27(a)-(f). That is, the moving speed of the cutting position determined in step S304 is shown in (c) of Table 1, and the gyro moving speed calculated in step S4002 is shown in (d) of Table 1. Also, the expected value calculated in step S4006 is shown in Table 1 (e).

As shown in Table 1, the subroutine of the clipping range correction process in FIG.

At first, the user is looking forward (front), so the gyro movement speed calculated in step S4002 is approximately 0°/sec. That is, in step S4003, it is determined that the gyro movement speed is not decelerated, and the process proceeds to step S4004. In this case, since the position of the face does not change, the moving speed of the clipping position is also 0°/sec. Also, the subtraction amount calculated in step S4004 is 0°/sec.

After about 1 second, the user starts to turn to the right, but due to the time lag caused by the low-pass filter, the movement speed of the cutout position is still 0°/second, as shown in FIG. 27(c). On the other hand, as shown in FIG. 27(d), the camera body 1 has not yet moved, that is, the gyro movement speed is also approximately 0°/sec. Therefore, the amount of subtraction calculated in step S4004 is also 0°/sec, like when the user is still standing still in front.

When the user turns to the right further and after about 2 seconds, as shown in FIG. On the other hand, as shown in FIG. 27(d), the camera body 1 has not yet moved, that is, the gyro movement speed is still approximately 0°/sec. Therefore, the subtraction amount calculated in step S4004 is 10°/sec.

The user turns to the right further, and after about 4 seconds, the user's body also begins to turn to the right. That is, as shown in FIG. 27(d), the orientation of the camera body 1 changes, so the gyro movement speed is 10°/sec. As the body begins to rotate, the actual angular velocity of the face decelerates by the relative velocity between the camera body 1 and the direction of the face, as shown in FIG. 27(b). The movement speed of the cutout position shown in c) is still 10°/sec at this time. Therefore, considering this time lag, the subtraction amount calculated in step S4004 is 10°/sec.

After about 5 seconds when the user turns further to the right, the gyro movement speed continues to be 10°/sec (Fig. 27(d)), but the movement speed of the cutting position shown in Fig. 27(c) slows down. 0°/sec. Therefore, the subtraction amount calculated in step S4004 is -10°/sec.

Although not shown in FIG. 27, when the user finishes turning to the right in about 6 seconds, the gyro movement speed becomes 0°/second, and in this case the process proceeds to step S4006. In this case, the sum of the subtraction amounts calculated so far and stored in the primary memory 103 is +10°/sec. The expected value is calculated by allocating the sum of this subtraction amount so that the change in the movement speed of each clipping position stored in the primary memory 103 in the past fixed period is constant. Here, the movement speed of the cutting position shown in FIG. , 10°/sec and 0°/sec. Therefore, all the expected values for the period of 2 to 6 seconds are set to 10°/second in order to keep the change in the moving speed of each cutout position constant (here, no change) during this period.

In the present embodiment, for the sake of simplification of explanation, the explanation is given every second, but usually the frame rate of moving image pickup is 24 to 60 fps. On the other hand, face orientation detection and gyro detection often do not need to be performed 60 times per second, so it is preferable to change the timing of face orientation detection processing and clipping range correction processing from the timing of imaging. For example, imaging is performed at 60 fps, but the timing of performing face direction detection processing and clipping range correction processing may be performed at 10 fps without any problem, and can be changed as appropriate in consideration of usage, power consumption, and the like.

As described above, in this embodiment, when the observation direction is greatly moved, the moving speed of the field of view as a moving image changes at the point when the movement of the face and the movement of the body (camera body) are combined. Thus, an example is shown in which the moving speed in the observation direction can be made constant.

In this embodiment, an example of cropping an ultra-wide-angle image according to the viewing direction has been described, but the present invention is not limited to this. For example, the overall control CPU 101 (imaging direction changing means) may change the imaging direction of the imaging section 40 according to the observation direction. However, in this case, a mechanism (driving means) for mechanically driving the imaging direction of the imaging unit 40, specifically, the orientations of the imaging lens 16 and the solid-state imaging device 42 in the yaw direction and the pitch direction by a method not shown is provided on the camera body. must be set to 1.

In this embodiment, smoothing of the face direction detection result is shown as an example, but when the general control CPU 101 (anti-vibration means) performs the anti-vibration control described in the first embodiment, tracking in the face direction is delayed. Therefore, it is preferable to perform similar processing.

(Example 5)
In the fifth embodiment, the visual field of the user and the secondary recorded video (hereinafter referred to as "recorded video") are caused by the parallax due to the positional difference between the position of the user's eyes and the mounting position of the photographing/detecting unit 10. A method for reducing the difference in is described with reference to FIGS.

Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of the fifth embodiment, the same reference numerals are used for the configurations that are the same as those of the camera system of the first embodiment, and redundant explanations are omitted, and details of different configurations are added each time. to explain.

First, in order to facilitate understanding, the difference between the user's field of view and the recorded image that occurs in the first embodiment will be described.

FIG. 29 is a schematic diagram for explaining the relationship between the field of view of the user 5010 and the target field of view when the object 5020 at a short distance is the object of observation in the first embodiment.

FIG. 29(a) is a schematic diagram showing an image 5900 including an observation target 5020 reflected on the solid-state imaging device 42, and FIG. 29(b) is a schematic diagram showing the positional relationship between the user 5010 and the observation target 5020. is.

As shown in FIG. 29B, when the observation target 5020 is below the height of the user's eyes 5011, the face direction 5015 of the user 5010 faces downward. At this time, an observation target 5020 against a background such as a floor (not shown) appears in the field of view of the user 5010 .

In the case of the first embodiment, an observation direction 5040 (FIG. 29B) parallel to the user's face direction 5015 detected by the face direction detection unit 20 is set as the recording direction. Therefore, as shown in FIG. 29B, when the subject at a short distance is the observation target 5020, there arises a problem that an area not including the observation target 5020 is set as the target visual field 5045. FIG.

In such a case, even if the background reflected in the field of view of the user 5010 (floor, not shown) is different from the background captured by the imaging/detection unit 10 (ceiling, not shown), the recording direction is the observation direction 5040. Instead, an area 5035 including an observation target 5020 should be set in a direction 5030 that is the target field of view.

The above problem is caused by parallax due to the positional difference between the position of the eye 5011 of the user 5010 and the mounting position of the photographing/detecting unit 10 . Therefore, in this embodiment, parallax correction mode processing is executed to appropriately adjust the recording direction set based on the face direction of the user 5010 according to the parallax.

FIG. 31 is a block diagram showing the hardware configuration of the camera body 1 according to this embodiment.

The hardware configuration of the camera body 1 according to this embodiment differs from that of the camera body 1 according to Embodiment 1 shown in FIG. Although the arrangement position of the distance measuring sensor 5100 in the camera body 1 is not particularly limited, in this embodiment, the distance measuring sensor 5100 is provided on the outer edge of the stop switch 15 as shown in FIG.

Distance sensor 5100 is a sensor that measures the distance to an object. Note that the configuration of the ranging sensor 5100 is not particularly limited. In this example, the ranging sensor 5100 is an active type sensor that projects infrared rays, lasers, millimeter waves, or the like onto an object and measures the distance to the object from the reflection. Also, the distance measuring sensor may be a passive type sensor that measures the distance to an object based on the phase difference of the light rays that have passed through the imaging lens 16 .

The distance measurement sensor 5100 is connected to the overall control CPU 101 and controlled by the overall control CPU 101 .

FIG. 32 is a schematic diagram of the relationship between the user, the calibrator 850, and the target visual field 5080 at the time of calibration including parallax correction mode processing in this embodiment.

FIG. 32(a) is a schematic diagram showing an image 5900 including the calibrator 850 projected on the solid-state imaging device 42, and FIG. 32(b) is a schematic diagram showing the positional relationship between the user 5010 and the calibrator 850. .

A target field of view 5080 in FIG. 32A is a target field of view when calibration including parallax correction processing described later has not been performed and the face direction 5015 detected by the face direction detection unit 20 faces the front. .

On the other hand, the target field of view 5090 in FIG. 32( a ) is the target when calibration including parallax correction processing described later has been performed and the face direction 5015 detected by the face direction detection unit 20 faces the front. field of view.

FIG. 33A is a flowchart of parallax correction mode processing that is part of the preparation processing in step S100 of FIG. 7A according to this embodiment. Details of this process will be described below with reference to FIGS.

In the preparatory operation process of step S100 in FIG. 7A related to this process, when the parallax correction mode is activated by the user 5010 operating the calibrator 850 (step S5101), the display device control unit 801 displays the positioning index 851 (step S5102).

Subsequently, the display device control unit 801 instructs the user on the position (specified position) where the calibrator 850 should be held. Specifically, the display device control unit 801 instructs the user 5010 to hold the positioning indicator 851 in front of him at eye level by displaying an instruction display 855 similar to that in FIG. 22A (step S5103). ).

When the instruction display 855 is performed, the user 5010 holds the calibrator 850 over the specified position instructed in step S5103 and directs the face direction 5015 toward the positioning index 851 (front). At this time, the user 5010, the positioning index 851, and the photographing/detecting section 10 are in a positional relationship as shown in FIG. 32(b).

After that, when the display device control unit 801 determines that the user has viewed the positioning index center 852 in the center of the field of view, the distance 5050 (FIG. 32(b)) between the photographing/detecting unit 10 and the positioning index 851 is (step S5104).

Subsequently, the overall control CPU 101 detects the horizontal axis 5060 of the photographing/detecting unit 10 by the angular velocity sensor 107 (attitude detecting means) (step S5105). Thereby, the horizontal position 5065 of the image 5900 (FIG. 32(a)) projected on the solid-state imaging device 42 is specified.

Also, in step S5105, the overall control CPU 101 acquires the distance 5855 between the center of the positioning index 851 on the image 5900 and the horizontal position 5065 (FIG. 32(a)). After that, the overall control CPU 101 (angle calculating means) calculates an angle 5055 (FIG. 32(b)) formed between the horizontal axis 5060 and the direction of the positioning index 851 viewed from the imaging/detecting section 10. FIG. This angle 5055 is calculated using the distance 5855 and information about the relationship between a point on the image 5900 and the angle of incidence of the light imaged at that point. This information is stored in memory (eg, internal non-volatile memory 102).

After that, the overall control CPU 101 (vertical distance calculation means) calculates a vertical distance 5070 between the photographing/detecting unit 10 and the eye 5011 of the user 5010 using the distance 5050 and the angle 5055 calculated in step S5105 (step S5106). ), exiting this subroutine.

Here, a method of measuring the vertical distance 5070 between the photographing/detecting unit 10 and the eye 5011 of the user 5010 by a method different from that of the second embodiment has been described, but the present invention is not limited to this. For example, the vertical distance 5070 between the photographing/detecting unit 10 and the eye 5011 of the user 5010 may be measured by the method described in the second embodiment, or the user 5010 may directly input the value of the vertical distance 5070. Also good.

The calibration processing including the parallax correction mode processing in this embodiment is basically the same as the processing in steps S3101 to S3111 of FIG.

However, in step S3110, in addition to the processing described in the second embodiment, parallax correction is performed based on the vertical distance 5070 (FIG. 32(b)) calculated by the parallax correction mode processing of FIG. 33A. That is, calibration is performed so that the field of view of the user 5010 and the target field of view 125 match at infinity.

FIG. 33B is a flowchart of a subroutine of the recording direction/range determination processing of S300 described in FIG. 7A according to this embodiment. This process will be described below with reference to FIG. 34 as well. In addition, in FIG. 33B, the same reference numerals are given to the steps that overlap with those in FIG. 7D, and overlapping explanations will be omitted.

In FIG. 33B, first, the overall control CPU 101 acquires distance information of the photographable range (imaging area) by the distance measuring sensor 5100 (distance measuring means) (step S5301).

Next, the overall control CPU 101 (creating means) creates a defocus map 5950 (Fig. 34(a); distance map information) based on the distance information (measurement result by the distance measuring sensor 5100) obtained in step S5301 (step S5302).

A defocus map 5950 in FIG. 34( a ) represents a defocus map created when capturing a situation in which an observation target 5020 is floating in a room as shown in FIG. 34( c ). Here, in order to clearly show the distance information in the defocus map 5950, it is divided into six distance areas (1) to (6) from the shortest distance from the photographing/detecting unit 10 and expressed. However, in practice, the defocus map may be created steplessly.

Subsequently, the overall control CPU 101 calculates the direction of the observation target 5020 viewed from the photographing/detecting unit 10 from the defocus map 5950, the face direction 5015, and the vertical distance 5070 (FIG. 32(b)) (step S5303). . That is, parallax correction is performed with respect to the observation direction set based on the face direction.

After that, the process of steps S301 to S305 in FIG. 7D is performed, and this subroutine is exited.

By using the defocus map 5950 created in this way and the detection result of the face direction 5015, it is possible to calculate the direction of the observation target 5020 as viewed from the photographing/detecting unit 10. FIG. It should be noted that, due to the parallax described with reference to FIG. 29 , it is not possible to measure only the distance to the observation target 5020 with the distance measuring sensor 5100 without creating the defocus map 5950 .

The degree of influence of parallax described in this embodiment varies depending on the distance between the user 5010 and the observation target. That is, since the effect of parallax can be ignored for an observation target that is some distance away from the user 5010, even in the recording direction/range determination processing in the first embodiment, the video is clipped and recorded in the target field of view including the observation target. It becomes possible to For example, when the user 5010 observes the observation target 5021 (FIG. 34(c)) located in the medium distance area (5) which is at least a certain distance from the user 5010, parallax correction in the recording direction is not performed in step S5303. can be The target visual field 5043 (FIG. 34(b)) set according to the recording direction 5041 (observation direction) inferred based on the face direction 5016 detected by the face direction detection unit 20 also includes the observation target 5021. It is from.

On the other hand, according to the present embodiment, the distance range between the user 5010 and the observation target in which the observation target of the user 5010 can be included in the target visual field can be expanded to the near side compared to the first embodiment. . For example, assume that the user 5010 is observing an observation target 5020 (FIG. 34(a)) located in the close range area (1), which is close to the user 5010. FIG. In this case, in the first embodiment, the observation direction 5040 (recording direction) is analogized based on the face direction 5015 detected by the face direction detection unit 20 . The target field of view 5042 (FIG. 34(b)) set according to this observation direction 5040 does not include the observation target 5020 . However, in this embodiment, parallax correction is performed with respect to the observation direction 5040 in step S5303 of FIG. 33B, and a target visual field 5036 including the observation target 5020 is set according to the recording direction after this parallax correction. Therefore, an observation target, such as the observation target 5020, which is so close to the user 5010 that the influence of parallax cannot be ignored can be captured satisfactorily.

Further, according to this embodiment, it is possible to record the observation object located in the middle distance area (5) in the center of the target field of view. For example, when the user 5010 is observing the observation target 5021 (FIG. 34A) located in the middle distance area (5), if the parallax correction in the recording direction 5041 is not performed as in the first embodiment, A target visual field 5043 is set in which the observation target 5021 is positioned at the upper end. On the other hand, in this embodiment, parallax correction is performed with respect to the recording direction 5041 in step S5303 of FIG. 33B, and a recording area 5037 in which the observation target 5021 is positioned at the center is generated according to the recording direction after this parallax correction.

By performing the parallax correction of the present embodiment in this way, it is possible to capture the observation target at the center of the clipped image more than in the first embodiment.

In this embodiment, parallax correction is also performed during calibration, and after making the user's field of view and the recording area match at infinity, the closer the distance between the user and the observation target at the time of imaging, the more before and after correction. Parallax correction was performed to increase the deviation in the recording direction. However, the parallax correction of this embodiment may be performed for a finite position, for example, a subject at a far distance or a short distance from the position of the calibrator 850 with respect to the user in the calibration processing of the second embodiment.

(Example 6)
In Example 6, a method for determining a cutout range when calculation of the viewing direction fails will be described with reference to FIGS. 35, 36A, and 36B.

Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of Example 6, the same reference numerals are used for the same configurations as those of the camera system of Example 1, and overlapping descriptions are omitted, and details of different configurations are added each time. to explain.

In the first embodiment, as shown in FIG. 7A, the target visual field is set in the recording direction/range determination process in step S300 based on the observation direction calculated from the face direction detected by the face direction detection unit 20 in step S200. ing. However, there are cases where the face direction detection section 20 is covered by an obstacle such as a collar or hair, the face direction detection section 20 breaks down, or the face direction detection section 20 moves away from the user. In such a case, the face direction of the user cannot be acquired, and the image of the target field of view that the user wants to capture cannot be captured.

In Patent Document 1, when a second camera that captures an image of a user fails to detect the user, the fact that the user has not been detected is not stored in the history of the user's observation information up to that point, and the user is not detected again. User detection is performed. In addition, when capturing an image following the direction of the face, if the detection of the direction of the face fails, the image capturing direction is determined according to the situation, thereby capturing an image that does not greatly deviate from the user's intention.

On the other hand, in the present embodiment, if the user's face direction can be detected, the face direction detection unit 20 detects the face direction as in the first embodiment, and the observation direction calculated based on this is detected. The image of the target field of view in the recording direction based on is captured. On the other hand, when the user's face direction cannot be detected and the user's viewing direction cannot be calculated, an image of a target field of view that captures the user's intention is picked up. That is, in this embodiment, after the face direction detection process ends in step S200, the viewing direction determination process is executed before the recording direction/range determination process is executed in step S300. In this process, when the face direction detection unit 20 fails to detect the user's face direction, the user's intention is determined according to the situation, and the viewing direction is estimated. That is, the image of the target field of view in the recording direction is captured based on information other than the observation direction calculated from the face direction.

FIG. 35 is a flow chart of viewing direction determination processing according to this embodiment, which is executed by the overall control CPU 101 . This processing will be described below with reference to FIGS. 36A and 36B.

First, in step S6001, it is determined whether or not the face direction detection unit 20 has acquired the face direction. If the face direction has been acquired, the process advances to step S6004, and the overall control CPU 101 (mode switching means) switches the mode of this processing to the face direction mode (first imaging mode), and determines the face direction by the method described in the first embodiment. is determined as the recording direction. After that, exit from this subroutine.

On the other hand, if the face direction could not be acquired (NO in step S6001), the overall control CPU 101 (mode transition means) advances to step S6002 to transition to another mode, and determines whether there is an object tracked in the past. determine whether

Here, the determination processing in step S6002 will be described with reference to FIG. 36A showing the relationship between the observation direction detection state of the user for each frame and the captured image.

In FIG. 36A, n is the frame number of the image, θ is the horizontal movement angle of the user's face, and the user state is the positional relationship between the user and the observation target in each frame. Also, the overall image indicates a super-wide-angle image captured by the imaging unit 40 in each frame, and the captured image indicates an image secondary-recorded in each frame, and corresponds to the area indicated by the dashed line in the overall image.

In FIG. 36A, as shown in each user state screen, the user is observing an object indicated by "□" at the bottom of the screen as an observation target, and n=5, that is, the user's observation in the fifth frame. A case where the direction could not be detected is exemplified.

In this embodiment, the current frame is used as a reference, and the past four frames are set as the predetermined period. If an object that can be determined to be the same three or more times during this predetermined period is included in the captured image, it is determined that there is an object that has been tracked in the past.

As shown in FIG. 36A, when n=1 to 4, although the movement angle θ changes by +10°, the captured image does not include objects indicated by squares that can be determined to be the same subject. there is Therefore, when n=5, it is determined that there is an object that has been tracked in the past.

Note that the determination criteria in step S<b>6002 may be changed according to the face direction detection period and the accuracy of the face direction detection unit 20 .

Returning to FIG. 35, if it is determined that there is an object (the same object) that has been tracked within the past predetermined time (YES in step S6002), the process proceeds to step S6005.

In step S6005, the mode of this processing is switched to a pre-subject tracking mode (second imaging mode) in which the pre-subject direction is the recording direction, and after determining the recording direction so as to track the subject, the process advances to step S6008. As described above, in this embodiment, even if the face direction cannot be detected, if there is a subject that has been tracked in the past, the recording direction is determined by shifting to the pre-subject tracking mode. can be reflected in the video. Since the method of recognizing the subject in the captured image and the method of subject tracking detection by the overall control CPU 101 (subject recognition means) are well known, detailed description thereof will be omitted.

On the other hand, if it is determined that there is no previously tracked subject (NO in step S6002), the process advances to step S6003.

In step S6003, it is determined whether or not the subject registered in advance in the built-in non-volatile memory (subject registration means) has been detected in the latest captured image.

In this embodiment, the pre-registration of the subject is performed by the user specifying an image containing a person to be captured from among the images stored in the display device 800, and the display device control unit 801 identifying the characteristics of the selected person. This is done by recognizing and transmitting it to the overall control CPU 101 in the camera body 1 . Note that the subject detected in step S6003 is not limited to this, and may be, for example, a subject included in a captured image acquired at the readout completion timing or other detection timing. A pattern matching method is used to determine whether the pre-registered subject matches the subject on the latest captured image. Since the pattern matching method is publicly known, detailed description thereof will be omitted.

If it is determined that the pre-registered subject has been detected in the latest shot video (YES in step S6003), the process advances to step S6006.

In step S6006, the mode of this processing is switched to a registered subject mode (third imaging mode) in which the recording direction is the direction of the subject detected in step S6003, and the recording direction is determined so as to track the registered subject. Proceed to S6008.

On the other hand, if it is determined that the pre-registered subject has not been detected in the latest captured image (NO in step S6003), it is determined that the subject to be observed cannot be estimated, and the process advances to step S6007.

In step S6007, the overall control CPU 101 (angle of view changing means) sets the mode of this processing to the recording direction before the detection of the face direction fails, while comparing the angle of view to be imaged with the specified angle of view and changes it to a wide angle. Then, switch to the subject lost mode (fourth imaging mode). After that, the process proceeds to step S6008. Note that the recording direction in the subject lost mode may be continuously moved by the amount of change in the viewing direction before the detection of the face direction fails.

Here, the case where the process proceeds to step S6007 in which the subject is lost mode will be described with reference to FIG. 36B.

In FIG. 36B, a case where n=5, that is, the user's viewing direction could not be detected in the fifth frame will be described as an example.

In the example of FIG. 36B, the main subject has not been found for n=1 to 4, and the pre-registered subject has not been found for the captured image for n=5. Therefore, the observation direction of n=5 is shifted to the right direction on the full-screen image, which is the movement direction of n=1 to 4, with inertia. In addition, the angle of view to be cut out from the full screen image is changed to a wide angle.

In step S6008, if the recording direction is determined from a direction other than the face direction in any one of steps S6005 to S6007, the general control CPU 101 (notification means) displays an error (detection error) indicating that face direction detection has failed. Notify users. After that, exit from this subroutine. In this embodiment, the vibrator 106 of FIG. 5 is used to warn the user. However, the notification method in step S6008 is not limited to the method of this embodiment, and other notification methods such as a warning using the LED 17 or display by a terminal that cooperates with the camera body 1 such as the display device 800 may be used. .

As described above, in this embodiment, when the face direction cannot be detected, the recording direction and the angle of view are changed according to the situation. can be avoided.

That is, in the present embodiment, when the face direction cannot be detected, but the subject tracked in the past or the pre-registered subject can be detected, the subject is tracked. On the other hand, if such a subject cannot be detected, the angle of view is changed from the specified angle of view to a wider angle in order to prevent missing shots and facilitate re-detection of the subject.

As a result, it is possible to prevent an image not intended by the user from being picked up due to failure in detection of the face direction.

Although the processing of steps S6001 to S6008 is performed for each frame, the mode can be changed based on the mode determination information such as whether the face direction is acquired from the face direction detection unit 20 even after the mode is changed. For example, in this embodiment, when a preregistered subject is detected as a result of widening the angle of view in the subject lost mode, the mode shifts to the registered subject tracking mode in which the direction of the detected subject is the observation direction. In this case, the widened angle of view is returned to the specified angle of view.

Also, in this embodiment, the mode is changed by one determination, but the mode may be changed based on the results of a plurality of times according to the frame rate and the detection capability of the face direction.

(Example 7)
In the seventh embodiment, a method of determining the observation direction according to the accuracy (reliability) of face direction detection will be described with reference to FIGS. 37 to 40. FIG.

Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of the seventh embodiment, the same reference numerals are used for the configurations that are the same as those of the camera system of the first embodiment, redundant descriptions are omitted, and details of different configurations are added each time. to explain.

In the sixth embodiment, the mode for determining the viewing direction is switched depending on whether or not the face direction has been detected, thereby preventing imaging in a recording direction unintended by the user. On the other hand, when the direction of the user's face cannot always be stably detected as in Patent Document 1, the image may be captured at an angle of view not intended by the user. Here, as an example where the face direction cannot be stably detected at all times, as shown in FIG. There is a case where the detection accuracy of is lowered.

As shown in FIG. 37, when the user faces the right direction (FIGS. 37B and 37C), the chin and cheek There are more areas that are hidden by the body and shoulders. That is, the camera body 1 has the characteristic that the face area that can be used for detecting the face direction is narrowed depending on the face direction, and the detection accuracy is likely to be lowered. This characteristic greatly depends on the mounting position of the camera body 1 by the user.

Therefore, in this embodiment, the detection accuracy (reliability) of the face direction is calculated according to the mounting position of the camera body 1 and the detection result of the face direction, and when the reliability is high, more face directions are reflected in the observation direction. If the reliability is low, more information other than that is reflected in the viewing direction. As a result, the intention of the user can be picked up and captured.

FIG. 38 is a flow chart of observation direction determination processing when obtaining a face direction according to this embodiment, which is performed instead of the processing in step S6004 of FIG. This processing is executed by the general control CPU 101 (observation direction determining means).

First, in step S7001, the overall control CPU 101 (first observation direction calculation means, reliability calculation means) detects the face direction θ _n (first (observation direction) to calculate face direction reliability _Tn .

The face direction reliability _Tn is calculated as follows.

First, face direction θ _n is divided into three components of face direction θ _yaw , θ _pitch and θ _roll . Here, the face direction θ _yaw indicates a rotation component for moving the face left and right, the face direction θ _pitch indicates a rotation component for moving the face up and down, and the face direction θ _roll indicates a rotation component for tilting the head.

In this embodiment, it is assumed that the camera body 1 is attached to the user's clavicle and the face direction is detected from the lower part of the face.

FIG. 39 shows changes in the values of face direction θ _yaw and face direction reliability T _n , and shows that the more the face direction θ _yaw changes from the front, the more the face direction reliability T _n decreases. .

In this embodiment, the face direction reliability T _n is calculated using the expression 701. However, depending on the detection accuracy of the face direction by the face direction detection unit 20 and the detection frame rate, the previously calculated face direction reliability Tn may be calculated. It is also possible to use the weighted average of the values calculated by weighting the reliability. Further, when calculating _Tn , the accuracy of pattern matching, the mounting position, etc. may be weighted.

In addition, in this embodiment, the facial orientation reliability for predicting the viewing direction is calculated by Equation 701. FIG. However, the method for calculating face direction reliability is not limited to this. For example, the face direction reliability adjusted according to the mounting position that can be estimated by the calibration of the second embodiment may be used. Further, when detection accuracy is determined to be low during calibration, face direction reliability may be changed according to the accuracy. Furthermore, when machine learning is used to detect the face direction, the matching rate may be reflected in the face direction reliability.

In step S7002, the overall control CPU 101 obtains the angular velocity ω _n of the movement of the face. Specifically, the face direction θ _n and the face direction acquisition time t _n acquired from the face direction detection unit 20 at the time of the nth frame imaging, and the face direction θ _n− of the previous frame stored in the primary memory 103 are ₁ and its acquisition time t _n−1 , the angular velocity ω _n is obtained by the following equation 702.

In this embodiment, the angular velocity ω _n is calculated using the information of the current frame and one frame before, but the angular velocity may be calculated using one or more pieces of past information depending on the frame rate or the like.

In step S<b>7003 , the overall control CPU 101 (observation direction prediction means) predicts the current face direction from the past face direction transitions stored in the primary memory 103 . In this embodiment, with the current frame as a reference, the past four frames are set as a predetermined period. It is judged that the observation direction can be predicted from the face direction and angular velocity of the face. When performing this prediction, the predicted angular velocity ω _ave that is the weighted average of the angular velocities obtained from the past four frames is calculated by the following equation 703, and the predicted face direction θ _ave (second viewing direction) is calculated as follows: It is calculated by the expression 704. The operations of equations 703 and 704 correspond to the processes shown in FIGS. 40(a1) and (a2), respectively.

It should be noted that the length of the predetermined period used in step S7003 and the weighted average method may be changed according to the frame rate and the detection accuracy of the face direction detection unit 20. FIG.

In step S<b>7004 , the overall control CPU 101 predicts the observation direction using the internal information other than the information from the face direction detection unit 20 among the information stored in the primary memory 103 . Specifically, in this embodiment, it is determined from the subject detection history whether the subject is currently being tracked. If it is determined that the object is being tracked, a predicted observation direction θ _sub (second observation direction) based on the movement of the subject is calculated. In this embodiment, the current frame is used as a reference, and the past four frames are set as a predetermined period. is being tracked. The subject tracking determination may be made by changing the determination criteria according to the detection cycle and detection accuracy of the overall control CPU 101 . Since the subject tracking detection method is well known, detailed description thereof will be omitted.

In this embodiment, the internal information used for predicting the observation direction in step S7004 is the subject detection history, but the present invention is not limited to this. For example, according to the mounting position and performance of the camera body 1, information about the user's face reflected in the photographing unit 40 and information about the movement and posture of the camera body 1 detected by the angular velocity sensor 107 and the acceleration sensor 108 are used. to predict the viewing direction. Also, as in step S6006 of the sixth embodiment, if there is a pre-registered subject, the overall control CPU 101 (third observation direction prediction means) may be obtained as the predicted viewing direction θ _sub .

In step S7005, overall control CPU 101 stores face direction detection-related information in primary memory 103 as a history. Here, the face direction detection related information includes the angular velocity ω _n of the face movement generated in step S7002, the face direction reliability T _n calculated in step S7001, and the face direction θ _n detected by the face direction detection unit 20. , face direction acquisition time t _n , and information indicating the generation time point of each of these information.

In step S7006, the overall control CPU 101 determines whether or not the face direction reliability Tn calculated in step _S7001 is equal to or greater than a predetermined value. If the face orientation reliability Tn is equal to or greater than the predetermined value, it is determined that the reliability of the face orientation is high, and the process advances to step _S7009 .

In step S7009, the general control CPU 101 determines the face direction to be the current viewing direction _θ'n , and proceeds to step S7013.

On the other hand, if the face orientation reliability Tn calculated in step _S7001 is less than the predetermined value (NO in step S7006), the process proceeds to step S7007.

In step S7007, if the predicted face direction θ _ave can be estimated in step S7003 and |θ _n −θ _ave | is within a predetermined angle, the process advances to step S7010. In this embodiment, the determination is made with the predetermined angle set to π/8.

In step _S7010 , the overall control CPU 101 (first viewing direction prediction means) determines the current viewing direction _θ'n using _θn , _θave , and face direction reliability Tn. In this embodiment, the current observation direction θ' _n ' is obtained by the following equation 705, and the process proceeds to step S7013. The calculation of expression 705 corresponds to the processing shown in FIG. 40(b). As shown in FIG. 39, the smaller the absolute value of the face direction θ _yaw angle, the greater the face direction reliability _{Tn .} ' _n largely reflects the face direction θ _n as shown in equation 705 . On the other hand, when the absolute value of the angle of the face direction θ _yaw is large, the current observation direction θ′ _n contains a lot of other information (predicted face direction θ _ave ) other than the face direction θ _yaw as shown in Equation 705. reflected.

If the above condition is not satisfied in step S7007, the process proceeds to step S7008, and if the condition that the predicted viewing direction θ _sub can be estimated and |θ _n −θ _sub | is within a predetermined angle is satisfied, The process advances to step S7011. As in step S7010, in this embodiment, the predetermined angle is set to π/8 for determination.

In step S7011, the overall control CPU 101 (second viewing direction prediction means) determines the current viewing direction _θ'n using the face direction _θn , predicted viewing direction _θsub , and face direction reliability _Tn . . In this embodiment, the current observation direction _θ'n is obtained by the following equation 706, and the process proceeds to step S7013. As in step _S7010 , as shown in FIG. 39, the face direction reliability Tn increases as the absolute value of the angle of the face direction θ _yaw decreases. Therefore, when the absolute value of the angle of the face direction θyaw is small, the face direction _θn is largely reflected in the current viewing direction _θ′n as shown in Equation 706. On the other hand, when the absolute value of the angle of the face direction θ _yaw is large, the current viewing direction θ′ _n contains a lot of other information (predicted viewing direction θ _sub ) other than the face direction θ _yaw as shown in Equation 706. reflected.

If the above condition is not satisfied in step S7008, it is determined that the currently reliable observation direction cannot be obtained, and the process proceeds to step S7012.

In step S7012, the current viewing direction _θ'n is determined by moving the previous viewing direction θ'n _-1 with inertia based on the displacement of the past viewing direction, and the angle of view is set to a specified value. , and advances to step S7013. This reduces the possibility that the user will miss the intended subject.

In this embodiment, the method of calculating the current observation direction _{θ′n is} switched according to the face direction reliability Tn and the subject detection state, but the present invention is not limited to this. For example, when calculating each of the predicted face direction θ _ave and the predicted observation direction θ _sub , their reliability (prediction direction reliability) is also calculated, and the calculated observation direction θ ' _n may be corrected.

In addition, if each calculated reliability is less than a predetermined value, it is likely that the user will miss the intended subject. It is preferable to In that case, the process may proceed to step S7012. Moreover, after that, if any of the calculated degrees of reliability becomes larger than a predetermined value, the angle of view is preferably returned to the specified angle of view.

By the processing of FIG. 38, when the face direction reliability T _n is high, the face direction θ _n is determined as the current observation direction θ′ _n . On the other hand, when the face direction reliability T _n is low, the current observation direction θ′ _{n (} recorded direction), and depending on the situation, widen the angle of view.

That is, in the present embodiment, when the face direction reliability _Tn is low and the accuracy of face direction detection is expected to be low, the predicted face direction θ _ave and the predicted observation direction θ _sub are used to detect the face direction. It is possible to prevent an image that is not intended by the user from being picked up due to detection failure.

(Example 8)
In the eighth embodiment, a method for mounting the camera body 1 in a stable position will be described with reference to FIGS. 41-45.

Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of the eighth embodiment, the same reference numerals are used for the configurations that are the same as those of the camera system of the first embodiment, and redundant descriptions are omitted, and details of different configurations are added each time. to explain.

First, angle adjustment of the connecting portions 80L and 80R (neck hanging means) will be described.

FIG. 41 is an enlarged view showing a side view of the photographing/detecting unit 10. As shown in FIG. In the following description, the left connection portion 80L will be described as an example, but the right connection portion 80R is similarly adjusted.

FIG. 41(a) is a diagram when the connection portion 80L is in the standard position Ax0, and FIG. 41(b) is a diagram in which the connection portion 80L is rotated by an angle θA1 with respect to the standard position Ax0 around the rotation axis OA. It is a figure when it exists in position Ax1. FIG. 41(c) is a schematic diagram showing a mechanical mechanism inside the angle holding portion 81L that can be visually recognized when the exterior of the angle holding portion 81L is removed.

As shown in FIG. 41(c), an angle adjusting mechanism 8100 (neck hanging angle adjusting means) is arranged inside the angle holding portion 81L.

The angle adjusting mechanism 8100 includes an angle adjusting cam 8101 for adjusting the angle of the angle holding portion 81L with respect to the photographing/detecting portion 10, and a locking member 8102 for locking the angle adjusting cam 8101. The rotation axis OA of the angle holding portion 81L coincides with the center of the angle adjusting cam 8101. As shown in FIG.

The locking member 8102 is a member biased against the angle adjustment cam 8101 by a spring (not shown), but the bias is released while the angle adjustment button 85L (FIG. 2F) is pressed. Move away from the angle adjustment cam 8101 . That is, the angle holding portion 81L of the connection portion 80L is rotatable with respect to the photographing/detecting portion 10 only while the angle adjustment button 85L is pressed.

The user rotates the angle holding portion 81L with respect to the imaging/detecting portion 10 while pressing the angle adjustment button 85L, thereby moving the connection portion 80L from the standard position Ax0 (FIG. 41A) to the position Ax1 (FIG. 41A). (b)) can be adjusted.

As a mechanism for holding the angle of the angle holding portion 81L with respect to the photographing/detecting portion 10, this embodiment uses a stepped adjustment mechanism using an angle adjustment cam 8101 and a locking member 8102. A mechanism that enables stepless adjustment using the may be used.

In this embodiment, the user presses the angle adjustment button 85L and rotates the angle holding portion 81L, but the present invention is not limited to this. For example, there is no need for the angle adjustment button 85L, and the angle holding portion 81L is rotated when an external force equal to or greater than a certain threshold value is applied to the angle holding portion 81L. It is also possible to use one that uses sliding resistance or one that uses sliding resistance.

FIG. 42 is a side view showing how the user wears the camera body 1. FIG.

FIG. 42(a) is a diagram showing a user wearing the camera body 1 in a state where the connection portion 80L is at the standard position Ax0 and the band portion 82L is adjusted to be long. FIG. 42(b) is a diagram showing a user wearing the camera body 1 in a state where the connection portion 80L is at the standard position Ax0 and the band portion 82L is adjusted to be short. FIG. 42(c) is a diagram showing a user wearing the camera body 1 in a state where the connecting portion 80L is at the position Ax1 and the band portion 82L is adjusted to be short.

As shown in FIGS. 42(a) and 42(c), if the relationship between the position of the connecting portion 80L and the length of the band portion 82L is suitable for the user, the imaging lens 16 faces the front of the user. Turn. On the other hand, as shown in FIG. 42(b), if the relationship between the position of the connecting portion 80L and the length of the band portion 82L is not suitable for the user, the imaging lens 16 does not face the front of the user. . In the case of FIG. 42(b), the optical axis of the imaging lens 16 faces upward.

Since the position of the connecting portion 80L is adjustable in this way, the user can adjust the position of the imaging lens 16 so that the optical axis of the imaging lens 16 is substantially parallel to the user's natural line-of-sight position. The camera body 1 can be attached. Of course, even if the mounting position of the camera body 1 suitable for the user is such that the optical axis of the imaging lens 16 is in the horizontal direction, it is possible to mount the camera body 1 appropriately as well.

Next, angle adjustment of the chest connection pads 18a and 18b will be described.

FIG. 43 is an enlarged view showing a side view of the photographing/detecting section 10 when the connecting sections 80L and 80R are hidden. In the following description, the left chest connection pad 18a is taken as an example, but the right chest connection pad 18b is similarly adjusted.

FIG. 43(a) is a diagram when the chest connection pad 18a is at the standard position Bx0, and FIG. 43(b) is a diagram in which the chest connection pad 18a is positioned at an angle θB1 with respect to the standard position Bx0 about the rotation axis OB. It is a figure when it exists in position Bx1 which rotated. FIG. 43(c) is a schematic diagram showing the internal mechanical mechanism of the imaging/detecting section 10 that can be visually recognized when the exterior of the imaging/detecting section 10 is removed.

As shown in FIG. 43(c), an angle adjustment mechanism 8200 (grounding angle adjustment means) is arranged inside the photographing/detecting section 10. As shown in FIG.

The angle adjusting mechanism 8200 comprises an angle adjusting cam 8201 for adjusting the angle of the chest connection pad 18 with respect to the imaging/detecting section 10 and a locking member 8202 for locking the angle adjusting cam 8201 . A point OB shown in FIGS. 43(a) to 43(c) is the center of rotation of the chest connection pad 18a.

The locking member 8202 is a member that is urged against the angle adjustment cam 8201 by a spring (not shown). away from That is, the chest connection pad 18 is rotatable with respect to the imaging/detecting unit 10 only while the angle adjustment button 8203 is pressed.

The user can adjust the position of the chest connection pad 18 from position Bx0 to position Bx1 by rotating the chest connection pad 18 with respect to the imaging/detection unit 10 while pressing the angle adjustment button 8203 . .

As a mechanism for holding the angle of the chest connection pad 18 with respect to the imaging/detection unit 10, this embodiment uses a stepped adjustment mechanism using an angle adjustment cam 8201 and a locking member 8202. A mechanism that enables stepless adjustment using the may be used.

In this embodiment, the user presses the angle adjustment button 8203 and rotates the chest connection pad 18, but the present invention is not limited to this. For example, there is no need for the angle adjustment button 8203, and the chest connection pad 18 is configured to rotate when an external force exceeding a certain threshold is applied to the chest connection pad 18. For example, a ball is used instead of the locking member 8202. It is also possible to use one that uses sliding resistance or one that uses sliding resistance.

FIG. 44 is a side view showing how the user wears the camera body 1 when the connection portion 80L is hidden.

FIG. 44(a) shows a state in which a user with an upright chest wears the camera body 1 with the chest connection pad 18a in the standard position Bx0. FIG. 44(b) shows a state in which a user whose chest is lying down wears the camera body 1 with the chest connection pad 18a in the standard position Bx0. FIG. 44(c) shows a state in which a user whose chest is lying down wears the camera body 1 with the chest connection pad 18a positioned at the position Bx1.

As shown in FIGS. 44(a) and 44(c), if the position of the chest connection pad 18a is suitable for the inclination of the user's chest, the chest connection pad 18a is positioned against the user's chest. grounded over a wide area. On the other hand, as shown in FIG. 44(b), if the position of the chest connection pad 18a is not suitable for the inclination of the user's chest, the chest connection pad 18a is slightly tilted against the user's chest. Ground only to a certain extent. As shown in FIG. 44(b), when the range in which the chest connection pad 18a contacts the user's chest becomes smaller, the photographing/detecting unit 10 can easily touch the user's body when the user moves his/her body. In contrast, it moves, and the captured image is greatly blurred.

Since the angle of the chest connection pad 18a is adjustable in this way, the user wears the camera body 1 so that the chest connection pad 18a is placed over a wide range on the user's chest. This makes it possible to suppress blurring of the shot image.

In this embodiment, the chest connection pad 18a is arranged in the imaging/detection section 10, but the same effect can be obtained even if it is arranged in the connection section 80L. In this case, for example, a mechanism similar to the angle adjustment mechanism 8100 shown in FIG. 41(c) is arranged inside the connection portion 80L as a mechanism for adjusting the angle of the chest connection pad 18a with respect to the connection portion 80L.

Next, the structures of the band portion 82L and the electric cable 84 will be described in detail.

As described in the first embodiment, the battery section 90 (power supply means) of the camera body 1 and the photographing/detecting section 10 are separate modules that are electrically connected via the electric cable 84 .

If the electric cable 84 and the band portion 82L are separated from each other, it is not preferable in terms of the aesthetics of the camera body 1, and it is also not preferable in terms of troublesome operation when the user wears it around his/her neck. Therefore, it is desirable that the band portion 82L and the electric cable 84 are integrated, but the structure is not limited to that shown in FIG. 2B.

FIG. 45 is a diagram showing a connection surface 83L, which is a cut surface of the band portion 82L and the electric cable 84 integrally formed therewith.

45(a) to (c) show the case where the electric cable 84 is made of a flexible printed circuit board (FPC), and FIGS. indicates if

In FIGS. 45(a) and 45(d), the electric cable 84 is embedded inside the band portion 82L when viewed from the connection surface 83L. In this case, the material of the band portion 82L is preferably an injection-moldable elastic member such as silicone rubber, elastomer, rubber, or plastic. As a method of manufacturing a part in which the band portion 82L and the electric cable 84 are integrated, there is a method of inserting the electric cable 84 during injection molding of the band portion 82L. In addition, a manufacturing method is adopted in which the band portion 82L is composed of two parts, and the band portion 82L composed of the two parts is integrated by adhesive or heat welding after sandwiching the electric cable 84. good too. However, as shown in FIGS. 45(a) and 45(d), it suffices if a part in which the band portion 82L and the electric cable 84 are integrated can be manufactured, and the manufacturing method is not limited to the above two manufacturing methods.

45(b), (c), and (e), the electric cable 84 is in contact with the outside of the band portion 82L when viewed from the connection surface 83L.

In FIG. 45(b), the band portion 82L when viewed from the connection surface 83L is not provided with a shape for integrating with the electric cable 84, and the electric cable 84 is simply adhered to the surface of the band portion 82L. Because of its structure, it can be manufactured at low cost. If the electrical cable 84 (FPC in this case) is on the exterior side, the exterior can be improved by painting the FPC or covering it with a film. Also, in the configuration of FIG. 45(b), when the electric cable is on the neck contact side of the user, the feeling of wearing can be improved by painting or covering the electric cable with a film.

45(c) and (e), the band portion 82L as viewed from the connection surface 83L is provided with a concave shape 83a for integration with the electric cable 84, and the electric cable is placed inside the concave shape 83a. 84 is placed. In this case, if the recessed shape 83a is provided on the user's neck contact side, the appearance of the camera body 1 is ensured. It is possible to keep the user's feeling of wearing well without performing any special treatment. Furthermore, if an appropriate design is made when the band portion 82 is produced, no additional cost is required to provide the concave shape 83a, which is favorable in terms of cost.

In FIGS. 45(f) and 45(g) as well, the electric cable 84 is embedded inside the band portion 82L when viewed from the connection surface 83L, similarly to FIGS. 45(a) and 45(d). FIG. 45(f) shows a configuration with one electric cable 84, and FIG. 45(g) shows a configuration with a plurality of electric cables 84. FIG. These configurations are characterized in that the cross-sectional area of the band portion 82L on the connection surface 83L is ensured, unlike FIGS. 45A and 45D. The cross-sectional area of the band portion 82L at the connection surface 83L affects the torsional rigidity and bending rigidity of the band portion 82L. Affects stability to stay stable in position. That is, the greater the cross-sectional area of the band portion 82L on the connection surface 83L and the stronger the torsional rigidity and bending rigidity, the more stable the photographing/detecting section 10 is. It should be noted that the protruding side of the electric cable 84 is preferably located on the exterior side in order to ensure a good wearing feeling. The shapes shown in FIGS. 45(f) and 45(g) are suitable from the standpoint of ensuring the rigidity of the band portion 82L, although the protruding shape of the electric cable 84 can be seen from the band portion 82L.

As described above, the shapes shown in FIGS. 45(c) and 45(e) are preferable from the viewpoint of the balance between aesthetic appearance and wearing comfort, but other shapes shown in FIG. 45 can be used when emphasis is placed on cost and rigidity. .

(Example 9)
In the ninth embodiment, a modification of the camera system including the camera body 1 will be described with reference to FIGS. 46A and 46B.

Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of the ninth embodiment, the same reference numerals are used for the configurations that are the same as those of the camera system of the first embodiment, and redundant descriptions are omitted, and details of different configurations are added each time. to explain.

In Example 1, a general smart phone was used as the display device 800, but there are many smart phones on the market, and their computing capabilities are diverse. For example, the display device 800 of Example 1 has relatively high computing power. For this reason, when the camera body 1 transfers an image in the recording direction cut out from the ultra-wide-angle image to the display device 800, information necessary for optical correction processing and vibration reduction processing is added to the image. Based on this information, the display device 800 performs distortion aberration correction and vibration reduction processing. On the other hand, the smartphone used as the display device 800 by the user may have too low computing power to perform these corrections. This embodiment assumes such a case.

The camera system of this embodiment includes a camera body 1 ′ including an imaging device and a display device 9800 having lower computing power than the display device 800 .

In the camera body 1′, when the processing (steps S100 to S600 in FIG. 7A) up to the primary recording processing of the video is completed, the processing of steps S800 and S900 is executed without performing the transfer processing to the display device 9800. . After that, in the camera body 1', a process of transferring the image for which the processes of steps S800 and S900 have been completed to the display device 9800 is performed.

On the other hand, in the display device 9800, the image from the camera body 1' is secondary-recorded as it is without performing the processing of steps S800 and S900.

The camera system of this embodiment will be specifically described below.

FIG. 46A is a block diagram showing the hardware configuration of a display device 9800 connected to a camera body 1' as an imaging device according to this embodiment.

In FIG. 46A, the same reference numerals are assigned to the same hardware configuration as the display device 800 according to the first embodiment shown in FIG.

The display device 9800 differs from the display device 800 in that it has a display device control unit 9801 instead of the display device control unit 801 and does not have the face sensor 806 .

The display device control unit 9801 is configured by a CPU having lower computing power than the CPU configuring the display device control unit 801 (FIG. 6). Also, the capabilities of the built-in nonvolatile memory 812 and the primary memory 813 may be lower than in the first embodiment.

FIG. 46B is a functional block diagram of the camera body 1'.

In FIG. 46B, the same reference numerals are given to the same functional blocks as those of the camera body 1 according to the first embodiment shown in FIG. 4, and redundant explanations are omitted.

The functional block shown in FIG. 46B has an optical correction/stabilization processing unit 9080 that performs optical correction processing and image stabilization processing, and each functional block is executed under the control of the overall control CPU 9101 instead of the overall control CPU 101. It differs from the functional block in FIG. Another difference is that one of the predetermined communication partners of the transmission unit 70 is the display device 9800 instead of the display device 800 .

That is, in this embodiment, the optical correction/stabilization processing unit 9080 of the overall control unit CPU 9101 performs optical distortion correction and stabilization processing using optical correction values and gyro data. Therefore, in comparison with the image file 1000 transferred by the transmission unit 70 to the display device 800 in the first embodiment, the image file after the optical distortion correction and image stabilization processing, which is transferred by the transmission unit 70 to the display device 9800 in the present embodiment, is the same. less data.

In addition, in the display device 9800, since the processing of steps S800 and S900 is not performed, the computing power can be kept lower than that of the display device 800. FIG. In addition, it is possible to view images captured by the camera body 1' on a simple display device 900 (appreciation unit) such as a smart watch.

(Example 10)
In the tenth embodiment, a modification of the camera system including the camera body 1 will be described with reference to FIGS. 47 and 48. FIG. Basically, this embodiment will be described as being derived from the first embodiment. For this reason, among the configurations of the camera system of the tenth embodiment, the same reference numerals are used for the configurations that are the same as those of the camera system of the first embodiment, and redundant explanations are omitted, and details of different configurations are added each time. to explain.

In Example 9, as the display device 9800 with low computing power, high performance was required for the camera body 1'. However, if the performance of the camera body is increased, the cost of the overall control unit CPU and its peripheral devices may increase, and heat may be generated due to the process load. Therefore, in the tenth embodiment, a configuration in which the calculation capacity on the camera body side is lowered and the calculation on the display device side is increased will be described.

FIG. 47 is a functional block diagram of a camera system including a camera body 1001 and a display device 1080 showing the configuration of this embodiment. The same components as in the functional block diagram of the camera body 1 and the camera body 1' according to the first embodiment shown in FIG. 4 and the ninth embodiment shown in FIG.

The functional block diagram shown in FIG. 47 includes a recording direction/angle of view determination unit 1083, an image cutout/development processing unit 1084 that cuts out and develops an image, and an optical correction/stabilization processing unit that performs optical correction processing and image stabilization processing. 4 and 46B in that 1085 is in the display device 1080 .

In the camera body 1001, there are a face image primary processing unit 1030 for processing the face image detected by the face direction detection unit 20, a main image primary processing unit 1050 for processing the main image taken by the photographing unit 40, and these. An image synthesizing unit 1055 for synthesizing images has been added, a recording direction/angle of view determining unit 1083 and an image clipping/developing processing unit 1084 have been moved to the display device 1080, and an image separating unit 1082 has been added. In addition, a receiving unit 1081 is added to the drawing, the description of which was omitted in the first and ninth embodiments in which the functions of the display device 1080 were simple.

The order of processing will be described using the flowchart of FIG. Note that a flow that performs the same or similar processing as the flow in FIG. 7A is given a number obtained by adding 10000 to the step number (that is, a number with "10" added to the upper two digits), and description thereof will be omitted. Also, as an aid to the explanation, in FIG. 48, which device shown in FIG. 47 is performing the step is described on the right side of each step. That is, steps S10100 to S10700 in FIG. 48 are executed by the camera body 1001, and steps S10710 to S10950 are executed by the display device 1080. FIG.

In FIG. 7A of Embodiment 1, the face orientation is detected in step S200 after the preparatory operation process of step S100. do in parallel. Next, in S10450, the two video data shot in S10200 and S10400 are combined. Various methods of combining are conceivable, but the two videos may be combined into one video file, or the data frames of the two videos may be associated so as not to cause a shift.

In this embodiment, the explanation will be continued based on an example in which two videos are made into one video file. At S10450, the primarily recorded combined image is wirelessly transmitted to display device 10180 at S10700.

The steps after step S10710 are executed by the display device 1080. FIG. In step S10710, the images combined in S10450 are again separated into face shot images and main shot images. Next, in step S10720, face direction detection processing is executed to analogize the viewing direction from the separated face shot video. The details of the face direction detection process are as described with reference to FIG. 7C, as in the first embodiment.

In step S10730, recording direction/range determination processing is executed. In step S10750, the recording direction and field angle information determined in step S10730 are used to cut out the image from the actual photographed image separated in step S10710, and recording area development processing is executed to develop the area. . In step S10800, optical correction processing is executed to optically correct the image developed in the recording area in step S10750. In step S10900, anti-vibration processing is performed.

Of course, even in this embodiment, the order of steps S10800 and S10900 may be reversed. In other words, image stabilization processing may be performed first, and then optical correction may be performed.

In step S10950, the display device control unit (moving image recording means) performs secondary recording to record the image for which the optical correction processing and image stabilization processing in steps S10800 and S10900 have been completed in large-capacity nonvolatile memory 814, and performs this processing. finish.

In this embodiment, in step S10700, by sending an image obtained by synthesizing the actual photographing image and the face photographing image, the processing in the camera body 1001 can be simplified, and the cost and heat generation can be reduced. . As in the first embodiment, the gyro data and attitude data output from the angular velocity sensor 107 and the acceleration sensor 108 may also be transmitted to the display device 1080 at the time of transfer in step S10700.

(Example 11)
In the eleventh embodiment, a modification of the camera system including the camera body 1 will be described with reference to FIGS. 49 and 50. FIG. Basically, this embodiment will be described as being derived from the first embodiment, but since the basic configuration is similar to that of the tenth embodiment, the configuration of the camera system of the eleventh embodiment among the configurations of the camera system of the eleventh embodiment The same reference numerals are used for the same configurations as those in the above, and overlapping explanations are omitted.

In the tenth embodiment, a configuration has been described in which the computing power on the camera body side is lowered and the computation on the display device 1080 side is increased. Although this configuration can reduce the load on the overall control CPU, the amount of data transmitted from the transmission unit 70 increases, resulting in an increase in power consumption and the problem of heat generation.

In addition, general control CPUs installed in recent camera bodies are being developed that include circuits specialized for image processing. For example, it is already possible to detect the direction of the face, which is required in this case, and it is becoming possible to realize it while suppressing the increase in cost and power consumption. Utilizing this, in this embodiment, the camera body 1101 detects a face, determines the recording direction and angle of view, transfers the main image to which the data is added to the display device 1180, and performs image cutout, development processing, and the like. This is done within the display device 1180 .

FIG. 49 is a functional block diagram of a camera system including a camera body 1101 and a display device 1180 showing the configuration of this embodiment. The same components as those in the functional block diagram of the camera body 1, the camera body 1', and the camera body 1001 according to the first, ninth, and tenth embodiments shown in FIG. omitted.

4 and 46B in that an image clipping/development processing unit 1184 that performs image clipping and development, and an optical correction/stabilization processing unit 1185 that performs optical correction processing and image stabilization processing are provided in the display device 1180. FIG. Further, in accordance with the fact that the image cutout/development processing unit has been moved to the display device 1180 side, in the present embodiment, the overall control CPU 101 is provided with up to the recording direction/angle of view determination unit 30, but the image cutout/development processing unit 50 not prepared. An information synthesizing unit 1150 is added in the camera body 1101 for synthesizing the recording direction/angle of view information with the main image output from the photographing unit 40 . In the display device 1180, as in the tenth embodiment, an image clipping/development processing unit 1184 has been moved, and an information separation unit 1182 has been added. is added to the figure.

The order of processing will be described using the flowchart of FIG. It should be noted that a step number plus 11000 (that is, a number with "11" added to the upper two digits) is assigned to a flow that performs the same or similar processing as the flow in FIG. 7A, and the description is omitted. 50, on the right side of each step, which device shown in FIG. Steps S 11710 to S 11950 are executed by display device 1180 .

In the tenth embodiment, after the face photographing in step S10200 and the main photographing in S10400 are performed in parallel, the two image data photographed in S10450 are combined. In S11400, the recording direction/range is determined and the recording direction/range data is output.

After that, in step S11450, the actual photographing data from S11300 executed in parallel and the recording direction/range data output in S11400 are synthesized.

Various methods of synthesizing the recording direction/range data are conceivable, but in the eleventh embodiment, the recording direction/range data is recorded as the metadata for each frame in the actual photographing data. This has the same configuration as the metadata in FIG.

The actual photographing data created in S11450 is primarily recorded in S11600, and wirelessly transmitted to the display device 11180 in S11700.

The steps after step S11710 are executed by the display device 1180. FIG. In step S11710, the video with metadata combined in S11450 is separated again into the recording direction/range data and the actually shot video.

Next, in step S11750, a recording area development process is executed to cut out an image from the actually shot image separated in step S11710 using the recording direction and angle of view information and to develop the area.

In step S11800, optical correction processing is executed to correct optical aberration for the image developed in the recording area in step S11750. In step S11900, anti-vibration processing is performed.

Of course, even in this embodiment, the order of steps S11800 and S11900 may be reversed. In other words, image stabilization processing may be performed first, and then optical correction may be performed.

In step S11950, the display device control unit (moving image recording means) performs secondary recording to record the image for which the optical correction processing and image stabilizing processing in steps S11800 and S11900 have been completed in large-capacity nonvolatile memory 814, and performs this processing. finish.

In this embodiment, in step S11450, timed metadata that combines the video of actual shooting and the recording direction/range data is transmitted. It is possible to reduce heat generation and reduce the load on the display device 1180 . As in the first embodiment, the gyro data and attitude data obtained from the angular velocity sensor 107 and the acceleration sensor 108 may also be transmitted to the display device 1180 at the time of transfer in step S11700.

(Example 12)
In the twelfth embodiment, a configuration in which the imaging direction is changed by mechanically driving the direction of the imaging unit will be described with reference to FIGS. 51A to 56. FIG. FIG. 51A is an external view of the camera body 1220 in this embodiment.

By assigning the same numbers to the parts that have already been described in the first embodiment, they mean the same functions, and the description in this specification is omitted. The camera body 1220 includes an imaging/detection section 1221 , connection sections 80L and 80R, and a battery section 90 .

FIG. 51B is a perspective view showing details of the imaging/detection unit 1221 which is a part of the camera body 1220. FIG. Photographing BR> The photographing section 1221 includes a main body 1210 , a yaw drive shaft 1201 , a yaw drive base 1202 , a pitch drive shaft 1203 and the photographing section 40 . A main body 1210 includes a power switch 11 , an imaging mode switch 12 , a face direction detection window 13 , a start switch 14 , a stop switch 15 and a yaw drive motor 1204 .

The yaw drive motor 1204 drives the yaw drive base 1202 in the yaw direction (horizontal direction) via the yaw drive shaft 1201 . Yaw drive base 1202 includes pitch drive motor 1205 . The pitch drive motor 1205 drives the imaging unit 40 in the pitch direction (vertical direction) via the pitch drive shaft 1203 .

The imaging unit 40 includes an imaging lens 16 and a solid-state imaging device 42 (not shown). The imaging lens 16 guides the light beam from the subject and forms an image of the subject on the solid-state imaging device 42 .

FIG. 51C is a perspective view showing a state in which the photographing unit 40 is rotated 30 degrees to the left, and FIG. 51D is a perspective view showing a state in which the photographing unit 40 faces downward by 30 degrees. As shown in FIG. 51C, by driving the yaw drive motor 1204, the yaw drive shaft 1201 and the rest rotate left and right, making it possible to change the direction of the photographing unit 40 in the yaw direction. As shown in FIG. 51D, by driving the pitch drive motor 1205, the tip of the pitch drive shaft 1202 rotates up and down, making it possible to change the direction of the imaging unit 40 in the pitch direction.

FIG. 52 is a functional block diagram of a camera body 1220 according to the twelfth embodiment. Here, a rough flow of processing executed by the camera body 1220 will be described using FIG. In the following, changes to FIG. 4 will be described, and descriptions of the same processes will be omitted by using the same reference numerals.

52, a camera body 1220 includes a face direction detection section 20, an imaging section driving section 1230, an imaging section 40, a developing section 1250, a primary recording section 60, a transmission section 70, and other control section 111. FIG. These functional blocks are executed under the control of the general control CPU 101 (FIG. 53) that controls the camera body 1220 as a whole.

The face direction detection unit 20 detects the face direction, infers the viewing direction, and passes it to the imaging unit driving unit 1230 . The imaging unit driving unit 1230 performs various calculations based on the observation direction estimated by the face direction detection unit 20 and the outputs of the angular velocity sensor 107 and the acceleration sensor 108, and drives the imaging unit 40 to determine the imaging direction and imaging angle of view. to change

The photographing unit 40 converts the light beam from the object into an image and transfers the image to the developing unit 1250 . The developing section 1250 (developing means) develops the image from the photographing section 40 and passes the image in the direction in which the user is looking to the primary recording section 60 . The primary recording unit 60 passes it to the transmitting unit 70 at the required timing. The transmission unit 70 wirelessly connects with the display device 800 (FIG. 1D), the calibrator 850, and the simple display device 900, which are predetermined communication partners, and communicates with them.

FIG. 53 is a block diagram showing the hardware configuration of a camera body 1220 according to the twelfth embodiment. Only differences from FIG. 5 of the first embodiment will be described below. A camera body 1220 in FIG. 53 includes a phase detection sensor 1206 and a motor drive circuit 1207 . A phase detection sensor 1206 detects the pitch and yaw phases of the imaging unit 40 and outputs them to the overall control CPU 101 .

The motor driving circuit 1207 is controlled by the overall control CPU 101 to drive the imaging section 40 in a desired direction at a desired driving speed.

How to use the camera body 1 and the display device 800 will be described below. FIG. 54 is a flow chart showing an overview of image recording processing according to this embodiment, which is executed in the camera body 1220 and the display device 800. As shown in FIG. As an aid to the explanation, in FIG. 54, which device shown in FIG. 52 is performing the step is described on the right side of each step.

In step S100, when the power switch 11 is turned on and the camera body 1 is powered on, the overall control CPU 101 is activated and reads out the activation program from the built-in non-volatile memory 102. FIG. After that, the overall control CPU 101 executes preparatory operation processing for setting the camera body 1 before imaging.

In step S200, the face direction detection unit 20 detects the face direction, and performs face direction detection processing for analogizing the viewing direction. This process is executed at a predetermined frame rate.

In step S12300, imaging unit driving unit 1230 performs imaging unit driving processing for calculating the driving amount of imaging unit 40 and performing drive control. The details of the imaging unit driving process will be described later with reference to FIG.

In step S400, the imaging unit 40 performs imaging and generates imaging data. In step S12500, development unit 1250 executes development processing for developing the imaging data generated in step S400. Details of the development processing will be described later with reference to FIG.

In step S600, primary recording section 60 (image recording means) executes primary recording processing in which the image developed in step S12500 is stored in primary memory 103 as image data. In step S700, the transmission unit 70 executes a transfer process to the display device 800 for wirelessly transmitting the video primarily recorded in step S600 to the display device 800 at a designated timing.

The steps after step S<b>800 are executed by display device 800 . In step S800, the display device control unit 801 executes optical correction processing for correcting optical aberrations on the image transferred from the camera body 1 in step S700.

In step S900, the display device control unit 801 performs image stabilizing processing on the image whose optical aberration has been corrected in step S800. Note that the order of steps S800 and S900 may be reversed. In other words, image stabilization processing may be performed first, and then optical correction may be performed.

In step S1000, the display device control unit 801 (moving image recording means) performs secondary recording to record the image for which the optical correction processing and image stabilization processing in steps S800 and S900 have been completed in the large-capacity nonvolatile memory 814. exit.

FIG. 55 is a flowchart of the subroutine of step S12300 of the imaging unit driving process described with reference to FIG. In step S12301, the overall control CPU 101 acquires the outputs of the angular velocity sensor 107, acceleration sensor 108, and phase detection sensor 1206. FIG.

In step S12302, based on the observation direction recorded in step S212 (FIG. 7C) and the outputs of various sensors acquired in step S12301, the overall control CPU 101 calculates the control amounts of the pitch drive motor 1205 and yaw drive motor 1204. do. The control at this time is feedback control and anti-vibration control for the target value, and may be calculated using a known control method.

The overall control CPU 101 controls the motor drive circuit 1207 (step S12303) based on the control value calculated in step S12302 to drive the pitch drive motor 1205 and the yaw drive motor 1204 (step S12304). is ended (step S12305).

FIG. 56 is a flow chart of the development processing subroutine of step S12500 described in FIG. The only difference from FIG. 7E is that Xi, Yi, WXi, and WYi are not obtained in step S502, and that after obtaining raw data for all areas in step S501, the process immediately proceeds to color interpolation in step S503.

If the mode selected by the imaging mode switch 12 is the moving image mode, the processing of steps S200 and S12300 and the processing of steps S400, S12500 and S600 are executed in parallel. In other words, the driving of the imaging unit 40 is continued based on the observation direction detection result while the imaging unit 40 continues imaging.

By performing photographing with the configuration and control described above, it is possible for the user to direct the photographing section 40 toward the user's observation direction and perform photographing without being conscious of the photographing action.

(Example 13)
The thirteenth embodiment detects the face direction using machine learning such as Deep Learning. In recent years, a machine learning model has been proposed that detects face orientation without detecting feature points such as eyes and nose (reference: Fine-Grained Head Pose Estimation Without Keypoints (2017)). By using these machine learning models, it is possible to detect the direction of the face by using an image that is placed at the collarbone position and photographed in the upward direction.

FIG. 57 is a block diagram showing the hardware configuration of the camera body according to this embodiment. The condensing lens 1311 is a lens that condenses reflected light from the face. The face image capturing device 1312 is composed of an image capturing driver, a solid-state image sensor, an image signal processing circuit, and the like, similarly to the image capturing unit 40, and captures a face image.

In Example 1, the image near the user's chin and the background are separated by the presence or absence of the reflected infrared rays 25 . On the other hand, when the face direction is detected using machine learning as in the thirteenth embodiment, an infrared LED lighting circuit is not required in the configuration of the face direction detection unit 20, and an image capturing operation using natural light, which is equivalent to the image capturing unit 40, is performed. part can be used.

The face direction calculation device 1313 performs filter calculation, which is the central processing of Deep Learning, at high speed. The face direction calculation device 1313 may be a dedicated processing device using ASIC or FPGA, or may be processed using the overall control CPU 101 .

By setting the parameters learned in advance in the face direction calculation device 13143 and inputting the image output from the face image capturing device 1312 to the face direction calculation device 13143, angle information representing the face direction can be obtained. To learn the parameters for detecting face direction, a large number of learning images are required, which are obtained by adding correct vertical and horizontal angle information to face images.

FIG. 58 is a schematic diagram showing an example of a learning image, (a) is 0 degrees in the horizontal direction and 0 degrees in the vertical direction, (b) is 30 degrees in the horizontal direction and 0 degrees in the vertical direction, and (c) is in the horizontal direction. A face image captured at 0 degrees and 30 degrees in the vertical direction is shown. For the learning image, the face is moved, for example, in increments of 10 degrees within the detection range of the face direction, and a plurality of images, for example, about 100, are taken at each angle and used.

For example, if the detection range of the face direction is -60 degrees to 60 degrees in the horizontal direction and -60 degrees to 50 degrees in the vertical direction, the vertical angle of the face will be changed to - while keeping the horizontal angle of the face constant. The horizontal range of the face is obtained by taking training images while changing the angle from 60 degrees to 50 degrees in increments of 10 degrees, and then taking similar pictures while changing the angle of the face in the horizontal direction by 10 degrees. Learning images are repeatedly photographed in the range of -60 degrees to 60 degrees.

In addition, learning images need to be captured to cover various conditions other than the angle of the face in order to deal with various users and situations. For example, it is necessary to select subjects that cover the assumed user's physique, age, and gender, and to prepare a wide range of training images so that differences in the assumed background, such as indoors and outdoors, can be absorbed. .

FIG. 59 is a flow chart showing processing for detecting face direction using machine learning. First, the photographing unit 1311 is used to photograph a face image (step 1331). Next, the photographed image is resized to a size for input to the face direction calculation device 1314 (step 1332). Next, the resized image is input to the face direction calculator 1314 to calculate the face direction (step 1333).

In machine learning processing such as Deep Learning, generally, in addition to processing results such as face direction, a reliability indicating the likelihood of the processing result is also calculated. At step 1334, it is determined whether or not this reliability is greater than or equal to a predetermined threshold. If the reliability is greater than or equal to the threshold in step 1334, the face direction calculated in step 1333 is set as a new face direction (step 1335). That is, the face direction is updated. If the confidence is less than or equal to the threshold at step 1334, the face orientation is not updated.

As described above, according to the thirteenth embodiment, the face direction can be detected using machine learning such as Deep Learning.

(Example 14)
A fourteenth embodiment detects the face direction using a ToF (Time Of Flight) camera. FIG. 60 is a block diagram showing the hardware configuration of the camera body according to this embodiment.

The ToF device 1411 has a light emitting source, and measures the distance to the subject using reflected light that is emitted from the light emitting source and reflected back to the subject. In this embodiment, the subject is the user's face.

There are mainly two types of distance calculation methods in ToF: the Direct ToF method, which measures the time difference between the time it takes for the light emitted from the light source to be reflected by the subject and return, and the Direct ToF method, which periodically emits light from the light source and calculates the There is an Indirect ToF method that measures the distance by detecting the phase difference between the . In this embodiment, any type of ToF can be used. The ToF device 1411 creates an image (distance image) representing the distance information by two-dimensionally mapping the measured distance information.

FIG. 61(a) is a schematic diagram showing a distance image when the ToF device 1411 is installed at the clavicle position and the upward direction is measured. In FIG. 61(a), a short distance portion is shown in white, and a long distance portion is shown in black. A distance image 1421 in FIG. 61(a) includes a face region 1421 from the base of the neck to the tip of the nose and an object 1422 reflected in the background. A face direction calculation device 1412 calculates a face direction based on the distance image created by the ToF device 1411 . In this embodiment, the face direction calculation device 1412 is assumed to be the general control CPU 101, but it is not limited to this, and for example, a dedicated CPU may be used.

FIG. 62 is a flowchart showing face direction calculation processing. The overall control CPU 101 first extracts a face portion from the distance image created by the ToF device 1411 (step 1431). When the ToF device 1411 is installed at the position of the clavicle for measurement, the face to be measured is located at a short distance, and other objects are located at a long distance. That is, by applying threshold processing to the distance image in FIG. 61(a) and making pixels whose pixel values are equal to or less than the threshold as black pixels, it is possible to extract only the face portion. The threshold may be a predetermined fixed value, or may be adaptively calculated according to the content of the image.

FIG. 61(b) is a schematic diagram showing an image obtained by subjecting the distance image of FIG. 61(a) to threshold processing and extracting a face portion. As shown in FIG. 61(b), the object 1422 in the background in FIG. 61(a) becomes a black pixel because it is equal to or less than the threshold, and only the face 1421 is extracted.

Next, the image in FIG. 61(b) is divided into regions according to the distance information (step 1432). FIG. 61(c) is a schematic diagram showing a region-divided image. FIG. 61C shows how the face area 1421 is divided into six areas, from the closest area 14211 to the farthest area 14216 .

Next, the neck position (neck rotation center) is extracted (step 1433). As described in the first embodiment, the neck position is in the region 14211 that is the closest, and the location is the center of the region 14211 on the left and right. Therefore, point 14217 in FIG. 61(d) is set as the neck position. The overall control CPU 101 then extracts the chin position (step 1434).

As described in Example 1, the tip of the chin is located in the area 14214 before the area 14215 where the distance increases rapidly. Therefore, the overall control CPU 101 sets a point 14218 in FIG. 61(d), which is the center of the region 14214 in the left and right direction and the farthest position from the neck position 14217, as the chin position.

Once the neck and chin positions are determined, the horizontal and vertical angles of the face are then detected and recorded as direction vectors (1435). The horizontal angle of the face can be detected by the method described in the first embodiment based on the neck position and the chin position. In addition, when acquiring a distance image using a ToF camera, once the chin position is determined, the distance is known, so the vertical face angle can also be detected by the method described in the first embodiment. can be done.

The general control CPU 101 detects the horizontal and vertical face angles and stores them in the primary memory 103 as the viewing direction vi of the user.

As described above, according to the fourteenth embodiment, the face direction can be detected using the ToF camera.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist. It also includes things such as not using some functions. Although some embodiments have been described in which not only the imaging direction but also the angle of view are changed, the example can also be implemented without changing the angle of view.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

1 camera body 10 photographing/detecting unit 11 power switch 12 imaging mode switch 13 face direction detection window 14 start switch 15 stop switch 16 imaging lens 17 LED
18 chest connection pad 19L, 19R microphone 20 face direction detection unit 30 recording direction/angle of view determination unit 40 photographing unit 50 image cutting/development processing unit 60 primary recording unit 70 transmission unit 80 connection unit 81 angle holding unit 82 band unit 83 connection Surface 84 Electric Cable 90 Battery Part 91 Charging Cable Insertion Port 92L, 92R Adjustment Button 93 Backbone Protection Notch 94 Battery 101 Overall Control CPU
102 built-in nonvolatile memory 103 primary memory 105 speaker 106 vibrator 107 angular velocity sensor 108 acceleration sensor 805 in-camera 806 face sensor 800 display device 801 display device control unit 807 angular velocity sensor 808 acceleration sensor 809 imaging signal processing circuit 850 calibrator 900 simple display device 901 action camera 902 head fixing accessory 903 omnidirectional camera 904 lens 905 shooting button 5100 ranging sensor 8100 angle adjusting mechanism

Claims (66)

Observation direction detection means that is worn on the user's body other than the head and detects the observation direction of the user;
an imaging means that is worn on the user's body and captures an image;
An image pickup apparatus, comprising image output means for outputting an image corresponding to the observation direction based on the image picked up by the image pickup means. 2. The imaging apparatus according to claim 1, wherein said observation direction detection means detects the observation direction of said user in three dimensions. The observation direction detection means outputs the horizontal observation direction of the user's face as an angle of the first detection direction, and outputs the vertical observation direction of the user's face as the first detection direction. 3. The imaging apparatus according to claim 1, wherein the output is as an angle of the second detection direction perpendicular to . The observation direction detection means includes infrared irradiation means for irradiating an infrared irradiation surface of the user with infrared rays, and infrared detection means for detecting the reflected light of the infrared rays reflected by the infrared irradiation surface. The imaging device according to any one of claims 1 to 3. The observation direction detection means obtains distance information in a plurality of distance areas of the infrared irradiation surface from the reflected infrared light detected by the infrared detection means, and detects the observation direction based on the distance information. 5. The imaging apparatus according to claim 4, characterized by: The observation direction detection means detects the positions of the neck rotation center and the tip of the chin of the user based on the distance information, and detects the observation direction from the positions of the neck rotation center and the tip of the chin. 6. The imaging device according to claim 5. The observation direction detection means is located in a distance area that is relatively close to the infrared detection means, among the plurality of distance areas, and is located at the center of the distance area in the left and right direction and at the closest distance. 7. The imaging apparatus according to claim 6, wherein a center of rotation is set. The observation direction detection means sets the chin position to a position closest to a distance area having a relatively large distance change and farthest from the center of neck rotation among the plurality of distance areas. 8. The imaging device according to item 6 or 7. The observation direction detection means outputs the horizontal observation direction of the user's face as an angle of the first detection direction, and outputs the vertical observation direction of the user's face as the first detection direction. output as the angle of the second detection direction perpendicular to
9. The imaging apparatus according to claim 8, wherein said observation direction detection means calculates the movement angle of said chin position around said neck rotation center as the angle of said first detection direction. The observation direction detection means outputs the horizontal observation direction of the user's face as an angle of the first detection direction, and outputs the vertical observation direction of the user's face as the first detection direction. output as the angle of the second detection direction perpendicular to
10. The imaging apparatus according to claim 8, wherein the angle of the second detection direction is calculated based on the light intensity of the reflected light at the chin position. 11. The imaging apparatus according to any one of claims 1 to 10, wherein said image output means cuts out and outputs an image corresponding to said observation direction from images picked up by said imaging means. The imaging device further comprises calibration means for performing calibration of the viewing direction detection means using a wirelessly connected calibrator,
The calibrator includes face detection means for detecting the user's face by projecting infrared rays,
12. The imaging according to any one of claims 1 to 11, wherein the observation direction detection means does not detect the observation direction while the face detection means projects the infrared rays. Device. metadata generating means for generating metadata including intra-video position information indicating the size and position of the video corresponding to the observation direction, for video of each frame output to the video output means;
13. The imaging according to any one of claims 1 to 12, wherein the video output means generates a video file in which the metadata and the video of each frame are encoded for each frame. Device. further comprising optical correction value obtaining means for obtaining an optical correction value based on the optical design of the imaging lens in the imaging means;
14. The imaging apparatus according to claim 13, wherein the optical correction value is included in the metadata. further comprising movement amount detection means for detecting the movement of the imaging device and acquiring the movement amount;
15. The imaging apparatus according to claim 13, wherein the amount of movement is included in the metadata. 16. The imaging apparatus according to claim 15, wherein said moving amount detection means is any one of an acceleration sensor for detecting acceleration, an angular velocity sensor for measuring angular velocity, and a magnetic sensor for measuring the direction of a magnetic field. A mobile device wirelessly connected to the imaging device according to any one of claims 13 to 16, comprising video file receiving means for receiving the video file;
a first extraction means for extracting the metadata from the video file;
a second extraction means for extracting a frame of video encoded together with the extracted metadata from the video file;
frame video correction means for correcting the frame video extracted by the second extraction means by using the metadata extracted by the first extraction means;
A mobile device, comprising moving image recording means for recording the frame image corrected by the frame image correcting means as a moving image. The observation direction detection means is face direction detection means for detecting the face direction of the user, and during calibration, the imaging means captures an image of the positioning index to acquire an image including the positioning index. , an acquisition/detection means for detecting the face direction by the face direction detection means;
position calculation means for calculating the position of the positioning index in the image captured during the calibration based on the shape of the positioning index included in the image acquired by the acquisition/detection means;
generating means for generating information indicating the relationship between the face direction detected by the acquisition/detection means and the position of the positioning index calculated by the position calculation means;
a first calibration means for calibrating the center position of the target field of view according to the face direction detected by the face direction detection means based on the information generated by the generation means; Item 17. The imaging device according to any one of Items 1 to 16. A calibrator wirelessly connected to the imaging device according to claim 18,
a first display means for displaying the positioning index;
a face detection means for detecting a face;
a second display for displaying a button for transmitting the calibration instruction to the imaging device when it is determined that the user is looking at the positioning index based on the detection result of the face detection means; and a display means for the calibrator. further comprising an angular velocity sensor;
2. The apparatus according to claim 1, further comprising: second calibration means for performing calibration of the center position of said target visual field by comparing information from said angular velocity sensor with information on the face direction detected by said face direction detection means. 20. The calibrator according to 19 above. The observation direction detection means and the imaging means are integrally configured,
The face detection means captures an image of the user's face including the imaging means worn by the user, and captures an image by the imaging device based on the size of the imaging means reflected in the face image. 21. A calibrator according to claim 19 or 20, further comprising vertical distance calculation means for calculating the vertical distance between the lens and the position of the user's eyes. The observation direction detection means is face direction detection means for detecting the face direction of the user,
The image pickup device includes a calculation means for calculating an angular velocity of the user's face based on the face direction detected by the face direction detection means, and a recording direction of the image output from the image output means based on the observation direction. and recording direction determining means for determining,
When it is determined that the user's face has moved at a predetermined angular velocity or more for a first predetermined period of time or more as a result of calculation by the calculation means, the recording direction determination means determines the movement in the detected face direction. 2. The imaging apparatus according to claim 1, further comprising delay means for changing said recording direction in a direction delayed with respect to said recording direction. Even if it is determined that the user's face has moved at a predetermined angular velocity or more for a first predetermined period of time or more as a result of calculation by the calculation means, during the imaging of the moving subject by the imaging means, 23. The imaging apparatus according to claim 22, wherein said recording direction determination means does not execute said delay means, and said image output means outputs said entire imaged image. The recording direction determination means terminates the delay means when the elapsed time after changing the delayed direction to the recording direction exceeds a second predetermined time, and determines the current direction detected by the viewing direction detection means. 24. The image pickup apparatus according to claim 23, wherein said recording direction is determined from the observation direction of . The recording direction determining means determines that the period during which the user's face moves at the predetermined angular velocity or more is less than the first predetermined time as a result of the calculation by the calculating means. 23. The imaging apparatus according to claim 22, wherein the means is not executed. When the image output by the image output means is switched from the image whose recording direction is changed by the delay means to the image whose recording direction is determined by the current observation direction detected by the observation direction detection means. 23. The imaging device according to claim 22, wherein a video effect is added to the image. movement amount detection means for detecting the amount of movement of the imaging device during moving image imaging by the imaging means; observation direction correction means for correcting the movement amount in the observation direction;
The observation direction correcting means includes:
when the movement amount detection means detects that the imaging device is accelerating, delaying the movement amount in the observation direction;
18. When the movement amount detection means detects that the imaging device is decelerating, the movement in the observation direction is accelerated so as to cover the delayed amount. , and any one of 22 to 26. 28. The image pickup apparatus according to claim 27, wherein said movement amount detection means detects said movement amount by comparing images of a plurality of frames obtained by moving image pickup by said image pickup means. 29. The image pickup apparatus according to claim 27, wherein said image output means outputs a part of the image picked up by said image pickup means, which is cut out according to said observation direction. driving means for driving the imaging direction of the imaging means in the yaw direction and the pitch direction;
30. The imaging apparatus according to any one of claims 27 to 29, further comprising imaging direction changing means for changing the imaging direction of said imaging means by said driving means according to said observation direction. 31. The apparatus according to any one of claims 27 to 30, wherein said observation direction correcting means corrects moving speed in said observation direction in a moving image output by said image output means so as to be substantially constant. imaging device. The observation direction detection means and the imaging means are integrally configured,
distance measuring means for measuring the distance from the imaging means to the imaging area;
creating means for creating distance map information of the imaging area from the result of measurement by the distance measuring means;
further comprising calculating means for calculating a direction of an observation target of the user as seen from the imaging means from the observation direction, the distance map information, and a vertical distance from the imaging means to the position of the user's eyes. 2. The imaging apparatus according to claim 1, wherein: attitude detection means for detecting a horizontal axis of the imaging means;
angle calculation means for calculating an angle between the detected horizontal axis and the direction of the external positioning index as seen from the imaging means;
From the angle calculated by the angle calculating means and the distance from the imaging means to the positioning index measured by the distance measuring means, the vertical distance from the imaging means to the eye position of the user is calculated. 33. The imaging apparatus according to claim 32, further comprising vertical distance calculating means for calculating the vertical distance. the calculated vertical distance, the observation direction detected by the observation direction detection means when the positioning index is at each specified position, and the distance measured by the distance measurement means from the imaging means to the positioning index 34. The image pickup apparatus according to claim 33, wherein the observation direction detected by said observation direction detection means is calibrated. a second recording direction determining means for determining a direction determined from information other than the viewing direction as the recording direction;
When the face direction detecting means can detect the face direction during moving image capturing by the image capturing means, the recording direction determining means determines the recording direction from the previous image capturing mode during the image capturing by the image capturing means. and the face direction detecting means cannot detect the face direction, the recording direction is determined by the second recording direction determining means during the imaging by the imaging means from the previous imaging mode. 23. The imaging apparatus according to claim 22, further comprising mode transition means for transitioning to one of the other imaging modes. further comprising subject recognition means for recognizing a subject from the video in the recording direction of the frame captured by the imaging means;
When the observation direction detection means cannot detect the observation direction and the same object is detected by the object recognition means within a predetermined time in the past, the mode transition means changes the direction in which the same object is followed. 36. The image pickup apparatus according to claim 35, wherein information other than the observation direction is set, and the immediately preceding image pickup mode is changed to a second image pickup mode which is one of the other image pickup modes. further comprising subject registration means for pre-registering a subject to be detected,
When the observation direction detection means cannot detect the observation direction, and the object recognition means detects the pre-registered object from the latest image, the mode transition means detects the registered object. 37. The image pickup apparatus according to claim 36, wherein the follow-up direction is set to information other than the observation direction, and the immediately preceding image pickup mode is shifted to a third image pickup mode which is one of the other image pickup modes. . When the observation direction detection means cannot detect the observation direction and the subject recognition means detects neither the same subject nor the pre-registered subject, the mode transition means performs the observation mode. One of the observation direction before it becomes undetectable by the direction detection means and the observation direction that continues to move with the amount of change up to that time is set as information other than the observation direction, and the imaging mode is changed from the previous imaging mode to the other imaging mode. 38. The image pickup apparatus according to claim 37, wherein the image pickup apparatus shifts to a fourth image pickup mode which is one of the modes. Further comprising a field angle changing means for changing the width of the field angle of the video output by the video output means,
39. The imaging apparatus according to claim 38, wherein, in said fourth imaging mode, said field angle changing means changes the field angle of the image in said recording direction to a wider angle than a prescribed field angle. 40. The imaging apparatus according to claim 39, wherein said mode transition means is continuously executed even after shifting to any one of said first to fourth imaging modes. When the mode transition means shifts from the fourth imaging mode to the first to third imaging modes, the angle of view of the image in the recording direction, which has been changed to the wide angle, is returned to a specified angle of view. 41. The imaging device according to claim 40, characterized by: further comprising notification means for notifying the user of a detection error in the observation direction when the observation direction detection means cannot detect the observation direction;
42. The apparatus according to any one of claims 39 to 41, wherein when said mode transition means shifts from said first imaging mode to said other imaging mode, said notification means notifies said user of said detection error. 1. The imaging device according to item 1. The observation direction detection means is
a first observation direction calculation means for detecting the face direction of the user and calculating a first observation direction from the detected face direction;
viewing direction prediction means for predicting a second viewing direction from information other than the detected face direction; reliability calculation means for calculating reliability of the first viewing direction;
if the reliability is greater than or equal to a predetermined value, determining the first observation direction as the observation direction; and if the reliability is less than the predetermined value and the second observation direction is determined to be reliable; , and viewing direction determination means for determining the viewing direction from the first viewing direction, the second viewing direction, and the reliability of the first viewing direction. Imaging device. 44. Any one of claims 1 to 16, 18, and 22 to 43, wherein the detection optical axis of said observation direction detection means and the imaging optical axis of said imaging means are arranged to face in different directions. 10. The image pickup device according to claim 1. 45. The image pickup apparatus according to claim 44, wherein the detection optical axis of said observation direction detection means is arranged to face the direction of said user's jaw from said observation direction detection means. 46. An image pickup apparatus according to claim 44 or 45, wherein an image pickup optical axis of said image pickup means is arranged so as to face a front direction of said user from said image pickup means. The imaging device, in which the observation direction detection means and the imaging means are integrated, is incorporated in a camera body,
47. Any of claims 44 to 46, wherein said imaging device has a horizontal overall length that is longer than a vertical overall length as viewed from the front of said user when said user is wearing said camera body. The imaging device according to any one of items 1 and 2. The camera body has a fixing means that is grounded on the user's body or clothes,
48. An image pickup apparatus according to claim 47, wherein said fixing means are arranged in the vicinity of left and right end portions of said image pickup apparatus when said user wears said camera body. 49. An imaging apparatus according to claim 48, wherein said camera body further comprises contact angle adjusting means for adjusting an angle at which said fixing means contacts said user. The imaging device is connected to neck hanging means for wearing the imaging device around the user's neck,
48. The imaging apparatus according to claim 47, wherein both ends of said neck hanging means are connected to said imaging apparatus in the vicinity of its left and right ends when said user wears said camera body. 51. The imaging device according to claim 50, wherein said neck hanging means comprises neck hanging angle adjusting means for adjusting an angle of said neck hanging means with respect to said imaging device. The neck hanging means has a cross-sectional shape that is not a perfect circle,
In a state in which the user wears the camera body, the distance between portions of the neck hanging means that are positioned symmetrically with respect to the imaging device decreases from bottom to top. 51. The imaging device according to claim 50. The imaging device is connected to power supply means by the neck hanging means,
53. The imaging device according to any one of claims 50 to 52, wherein the imaging device is arranged behind the user's neck when the user wears the camera body. the imaging device is connected by the power supply means and the power supply means,
54. An imaging apparatus according to claim 53, wherein said power supply means is arranged so as to pass through said neck hanging means. The imaging means has an imaging lens and an imaging element that converts an optical image formed by the imaging lens into RAW data, and the RAW data read from a predetermined region of the imaging element is used as an image captured by the imaging means. output and
3. The apparatus further comprises developing means for extracting data of an area narrower than the predetermined area from the RAW data and including a target field of view in the recording direction and a preliminary area therearound, and developing the data. 23. The imaging device according to 22. 56. An imaging apparatus according to claim 55, wherein said preliminary area is a preliminary area for image stabilization used for image stabilization processing. 57. The imaging apparatus according to claim 55, wherein said developing means sequentially switches the shape of said target visual field and the shape of said preliminary area according to said recording direction and optical characteristics of said imaging lens. 3. The image output means records the data cut out from the predetermined area and developed by the developing means as the image in the recording direction, and does not record the data not cut out from the predetermined area. 58. The imaging device according to any one of 55 to 57. 59. The image pickup apparatus according to claim 58, wherein said image output means transmits only the image in said recording direction among the images picked up by said image pickup means to an external viewing section. The viewing unit performs optical correction processing and image stabilization processing on the image in the recording direction,
60. The imaging apparatus according to claim 59, wherein said image output means adds information necessary for said optical correction processing and said image stabilizing processing together with the image in said recording direction. an observation direction detection step of detecting the observation direction of the user by means of observation direction detection means worn on the user's body other than the head;
an imaging step of imaging an image by imaging means worn on the user's body;
and an image output step of outputting an image corresponding to the viewing direction based on the image captured in the image capturing step. A control method for a mobile device wirelessly connected to the imaging device according to any one of claims 13 to 16,
a video file receiving step of receiving the video file;
a first extraction step of extracting the metadata from the video file;
a second extraction step of extracting a frame of video encoded together with the extracted metadata from the video file;
a frame video correction step of correcting the frame video extracted in the second extraction step using the metadata extracted in the first extraction step;
and a moving image recording step of recording the frame image corrected in the frame image correcting step as a moving image. A method for controlling a calibrator wirelessly connected to an imaging device according to claim 18,
a first display step of displaying the positioning index;
a face detection step for detecting a face;
(2) displaying a button for transmitting the calibration instruction to the imaging device when it is determined that the user is looking at the positioning index based on the detection result of the face detection step; and a display step. A computer-executable program that causes a computer to function as each means of the imaging apparatus according to any one of claims 1 to 16, 18, and 22 to 60. A computer-executable program causing a computer to function as each means of the portable device according to claim 17. A computer-executable program that causes a computer to function as each means of a calibrator according to any one of claims 19-21.

Images