JP2022140420A - Imaging apparatus, display unit, video transfer system, control method, and program - Google Patents

Abstract

To provide an imaging apparatus, a display unit, a video transfer system, a control method, and a program that can reduce the time required for image transfer.SOLUTION: In a camera system comprising a camera body 1 and a display unit 800, the camera body 1 has a photographing unit 40, an image quality conversion unit 2302 that converts a first video photographed by the photographing unit 40 into a second video with a smaller data size; an image recording unit 2301 that records the first video and the second video; and an entire control CPU that, when receiving editing information from the display unit 800, edits the first video based on the editing information and transmits the edited video to the display unit 800. When receiving the second video from the camera body 1, the display unit 800 sets the editing information based on the second video and transmits the editing information to the camera body 1 to acquire the edited first video from the camera body 1.SELECTED DRAWING: Figure 23

Description

The present invention relates to an imaging device, a display device, a video transfer system, a control method, and a program, and more particularly to an imaging device, a display device, a video transfer system, a control method, and a program used as an action camera.

Conventionally, when taking a picture with a camera, the photographer had to keep pointing the camera in the direction of the picture taking, so if the picture taking operation took their hands, they would not be able to do anything other than the picture taking action, or they would be forced to concentrate on the picture taking. Because of this, I was unable to concentrate on the experience of being there.

For example, in the imaging operation, the parent who is the photographer cannot play with the child while the child is being imaged.

In addition, in terms of concentrating on shooting, when shooting while watching sports, the photographer may not be able to support or remember the game content, and when focusing on watching sports, shooting may not be possible. problems arise. Similarly, when taking pictures during a group trip, the photographer cannot experience the same level of excitement as the other members, and giving priority to the experience causes a problem that the taking of pictures is neglected.

As a method to solve this kind of problem, by fixing the action camera to the head using a head fixation accessory and capturing the image in the observing direction, the photographer does not have to take the time to perform the image capturing operation. There is a way to take an image. In addition, by shooting a wide range with a omnidirectional camera, you can concentrate on the experience during the experience, and after the experience is over, cut out and edit the necessary video parts from the omnidirectional video taken, There is also a way to leave a video.

However, the former method requires the troublesome act of wearing a head fixing accessory to which the main body of the action camera 901 is fixed, as shown in FIG. 20(a). Also, as shown in FIG. 20(b), if the photographer attaches the action camera 901 to the head with the head-fixing accessory 902, the appearance is not good, and problems such as the hairstyle of the photographer being disturbed occur. Furthermore, the photographer may be concerned about the weight of the head fixing accessory 902 and the action camera 901 worn on the head, or may be concerned about the unattractive appearance of the third person. was Therefore, in the state shown in FIG. 20(b), the photographer cannot concentrate on the experience, or the photographer feels resistance to the state shown in FIG. 20(b). I had a problem with it being tough.

On the other hand, the latter method requires a series of operations such as image conversion and clipping position designation. For example, as shown in FIG. 21, an omnidirectional imaging camera 903 having a lens 904 and an imaging button 905 is known. The lens 904 is one of a pair of fish-eye lenses for half-sphere shooting arranged on both sides of the housing of the omnidirectional camera 903, and the omnidirectional camera 903 uses this pair of fish-eye lenses to perform omnidirectional imaging. Take ball shots. In other words, omnidirectional photography is performed by synthesizing the projection images of the pair of fisheye lenses.

22A and 22B are diagrams showing an example of the conversion work of the video imaged by the omnidirectional imaging camera 903. FIG.

FIG. 22A is an example of an image obtained by omnidirectional imaging by the omnidirectional imaging camera 903, and includes a photographer 906, a child 907, and a tree 908 as subjects. Since this image is obtained by synthesizing projection images of a pair of fish-eye lenses, the image is greatly distorted by the photographer 906 . Also, a child 907, which is the subject that the photographer 906 was trying to capture, has a body part in the peripheral part of the half-sphere optical system, so the body part is greatly distorted left and right and stretched. On the other hand, the tree 908 is an object positioned in front of the lens 904, so it is imaged without significant distortion.

In order to create an image of a field of view that people usually see from the image of FIG. 22(a), it is necessary to cut out a portion of the image, convert it into a plane, and display it.

FIG. 22(b) is an image obtained by cutting out an image located in front of the lens 904 from the image of FIG. 22(a). In the image of FIG. 22(b), a tree 908 is shown in the center in a field of view that people usually see. However, since the child 907 that the photographer 906 was trying to capture is not included in FIG. Specifically, it is necessary to set the cutting position to the left and 30° downward from the tree 908 as viewed in FIG. 22(a). FIG. 22(c) shows an image displayed after performing the clipping operation and then plane-converted. In this way, in order to obtain the image shown in FIG. 22(c), which the photographer intended to capture, from the image shown in FIG. must not. Therefore, there is a problem that the photographer can concentrate on the experience during the experience (during imaging), but the amount of work after that becomes enormous.

Therefore, Patent Document 1 discloses a technique of using a second camera that captures an image of a user in addition to a first camera that captures an image of a subject. In this technology, the user's moving direction and line-of-sight direction are calculated from the image captured by the second camera, the imaging direction of the first camera is determined, and the subject is estimated and captured from the user's preference and state. .

Further, in Patent Document 2, a sensor including a gyro and an acceleration sensor is mounted on the head to detect the observation direction of the photographer (user), and the sensor is detected from an imaging device separately mounted on the body or bag. An image recording system is disclosed that captures a detected viewing direction.

JP 2007-74033 A JP-A-2017-60078

However, in Patent Document 1, the second camera captures an image of the user from a position distant from the user. High optical performance is required for the second camera. In addition, there is a problem that the image processing of the image captured by the second camera requires a high arithmetic processing ability, and the apparatus becomes large-scale. Furthermore, even with that, the user's observation direction cannot be calculated precisely, so the subject cannot be accurately estimated from the user's preference and condition, and the image is different from the image desired by the user. There was a problem of capturing images.

In addition, in Patent Document 2, since the observation direction of the user is directly detected, the user must wear a sensor on the head, and the user must wear some device on the head as described above. I can't solve the hassle of the time. In addition, when the sensor consists of a gyro sensor or an accelerometer, a certain degree of accuracy can be obtained in detecting the relative observation direction, but the accuracy of detecting the absolute observation direction, especially in the horizontal rotation direction, cannot be obtained. had a problem.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging device, a display device, a video transfer system, a control method, and a program capable of shortening the time required for image transfer.

A video transfer system according to claim 1 of the present invention is a video transfer system comprising an image pickup device and a display device, wherein the image pickup device comprises image pickup means and a first image picked up by the image pickup means. into a second image having a data size smaller than that of the first image; recording means for recording the first image and the second image; first communication means; When editing information is received from the display device by one communication means, the first video is edited based on the received editing information, and the edited first video is sent to the display device by the first communication means. the display device receives the second image from the imaging device through the second communication means and the second communication means, and transmits the received second image to and transmitting the editing information to the imaging device by the second communication means, thereby transmitting the edited first video image by the second communication means. and acquisition means for acquiring from the imaging device.

According to a thirteenth aspect of the present invention, there is provided an image pickup apparatus that constitutes a video transfer system together with a display device, comprising: image pickup means; converting means for converting to a second image having a small data size; recording means for recording the first image and the second image; first communication means; and the display device by the first communication means image editing means for editing the first video image based on the received editing information, and transmitting the edited first video image to the display device through the first communication means when the editing information is received from characterized by having

A display device according to claim 14 of the present invention is a display device that constitutes a video transfer system together with an imaging device that captures a first video, and further includes: second communication means; Editing information setting means for receiving a second video having a data size smaller than that of the first video from the imaging device through the second communication means, and setting editing information based on the received second video. and acquiring means for acquiring the edited first image from the imaging device through the second communication means by transmitting the editing information to the imaging device through the second communication means. Characterized by

According to the present invention, the time required for image transfer can be shortened.

1 is an external view of a camera body including an imaging/detection unit as an imaging device according to Example 1. FIG. It is a figure which shows a mode that the user hung the camera main body. It is the figure which looked at the battery part in the camera main body from the back of FIG. 1A. 1 is an external view of a display device as a mobile device according to Example 1, which is configured separately from a camera body; FIG. It is the figure which looked at the imaging|photography / detection part from the front. FIG. 4 is a diagram showing the shape of the band portion of the connecting portion in the camera body; It is the figure which looked at the imaging|photography / detection part from the back side. It is the figure which looked at the imaging|photography / detection part from the top. FIG. 4 is a diagram showing the configuration of an infrared detection processing device arranged inside the photographing/detecting unit and below the face direction detection window in the camera body; It is the figure which looked at the state which the user hung the camera main body from the user's left side. It is a figure explaining the detail of a battery part. 3 is a functional block diagram of a camera body according to Example 1. FIG. 3 is a block diagram showing the hardware configuration of the camera body; FIG. It is a block diagram which shows the hardware constitutions of a display apparatus. 4 is a flow chart showing an outline of image recording processing according to the first embodiment, which is executed in a camera body and a display device; FIG. 7B is a flowchart of a subroutine of a preparatory operation process in step S100 of FIG. 7A according to the first embodiment; FIG. 7B is a flowchart of a subroutine of face direction detection processing in step S200 of FIG. 7A according to the first embodiment; 7B is a flowchart of a subroutine of a recording direction/range determination process in step S300 of FIG. 7A according to the first embodiment; 7B is a flowchart of a subroutine of recording area development processing in step S500 of FIG. 7A according to the first embodiment; FIG. 7B is a diagram for explaining the processing from steps S200 to S500 in FIG. 7A in the moving image mode; FIG. 10 is a diagram showing an image of a user seen through a face direction detection window; FIG. 10 is a diagram showing a case where an indoor fluorescent lamp is reflected as a background in an image of the user seen through the face direction detection window; When the user and the fluorescent lamp as the background shown in FIG. 8B are imaged by the sensor of the infrared detection processing device through the face direction detection window in a state where the infrared LED of the infrared detection processing device is not lit. is a diagram showing an image of . FIG. 8B is a diagram showing an image of the user shown in FIG. 8B and a fluorescent lamp as a background thereof formed by the sensor of the infrared detection processing device through the face direction detection window with the infrared LED turned on. is. FIG. 8B is a diagram showing a difference image calculated from the images of FIGS. 8C and 8D; FIG. FIG. 8E is a diagram showing a case in which the gradation of the difference image in FIG. 8E is adjusted by adjusting the scale to match the light intensity of the reflected infrared rays projected onto the face and neck of the user. FIG. 8F is a diagram in which symbols indicating each part of the user's body, double circles and black circles indicating the neck position and the chin position are superimposed on FIG. 8F. FIG. 8E shows a difference image calculated in the same manner as in FIG. 8E when the user's face is facing right; FIG. 8H is a diagram in which double circles and black circles indicating the neck position and the chin position are superimposed. FIG. 10 is a diagram showing an image of a user seen through the face direction detection window when the user faces 33° above the horizontal; When the user faces 33° above the horizontal, the difference image calculated by the same method as in FIG. It is a diagram. 4 is a timing chart showing lighting timings of infrared LEDs; It is a figure explaining a motion of a user's face of the up-down direction. FIG. 10 is a diagram showing a target field of view in a super-wide-angle image captured by the imaging unit of the camera body when the user faces the front; FIG. 11B is a diagram showing an image of the target field of view in FIG. 11A cut out from the ultra-wide-angle image; 4 is a diagram showing a target field of view in a super-wide-angle image when a user observes subject A; FIG. FIG. 11C is a diagram showing an image obtained by correcting distortion and shaking with respect to the image of the target field of view in FIG. 11C cut out from the ultra-wide-angle image. FIG. 11B is a diagram showing a target field of view in a super-wide-angle image when the user observes the subject A with a field angle set value smaller than that in FIG. 11C. FIG. 11C is a diagram showing an image obtained by correcting distortion and shaking with respect to the image of the target field of view in FIG. 11E cut out from the ultra-wide-angle image. FIG. 4 is a diagram showing an example of a target field of view in a super-wide-angle image; 12B is a diagram showing an example of a target field of view in a super-wide-angle image, which has the same field angle setting value as the target field of view in FIG. 12A but has a different observation direction. FIG. 12B is a diagram showing another example of a target field of view in a super-wide-angle image having the same field angle setting value as the target field of view of FIG. 12A but with a different observation direction. FIG. FIG. 12B is a diagram showing an example of a target field of view in a super-wide-angle image, which has the same observation direction as the target field of view of FIG. 12C but has a smaller field angle setting value. 12B is a diagram showing an example in which a preliminary area is provided around the target visual field shown in FIG. 12A; FIG. FIG. 12C is a diagram showing an example in which a spare area having the same image stabilization level as that of the spare area shown in FIG. 12E is provided around the target visual field shown in FIG. 12B. FIG. 12C is a diagram showing an example in which a spare area having the same image stabilization level as that of the spare area shown in FIG. 12E is provided around the target visual field shown in FIG. 12D. FIG. 4 is a diagram showing a menu screen for various settings of a moving image mode displayed on the display unit of the display device before the camera main body captures an image; 7B is a flowchart of a subroutine of primary recording processing in step S600 of FIG. 7A; FIG. 4 is a diagram showing the data structure of a video file generated by primary recording processing; FIG. 7B is a flowchart of a subroutine of transfer processing to the display device in step S700 of FIG. 7A; FIG. 7B is a flowchart of a subroutine for optical correction processing in step S800 of FIG. 7A; FIG. 18 is a diagram for explaining a case where distortion correction is performed in step S803 of FIG. 17; FIG. FIG. 7B is a flowchart of a subroutine of image stabilizing processing in step S900 of FIG. 7A; FIG. 1 is a diagram showing a configuration example of a camera fixed to the head using a conventional head fixing accessory; FIG. 1 is a diagram showing a configuration example of a conventional omnidirectional imaging camera; FIG. 22A and 22B are diagrams showing an example of conversion work of an image captured by the omnidirectional camera of FIG. 21; FIG. FIG. 10 is a functional block diagram of a camera body according to Example 2; 10 is a flowchart of video transfer processing of the camera system according to the second embodiment; FIG. 11 is a schematic diagram showing a transfer setting screen displayed on the display device according to the second embodiment; FIG. 11 is a functional block diagram of a camera body according to Example 3; 10 is a flowchart of video transfer processing of the camera system according to Example 3; FIG. 11 is a schematic diagram showing a cutout area edit screen displayed on the display device according to the third embodiment;

Preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

(Example 1)
1A to 1D are diagrams for explaining a camera system composed of a camera body 1 including a photographing/detecting unit 10 as an imaging device according to the present embodiment and a display device 800 configured separately from the camera body 1. FIG. . In this embodiment, the camera body 1 and the display device 800 are shown as separate bodies, but they may be constructed integrally. A user wearing the camera body 1 around the neck is hereinafter referred to as a user.

FIG. 1A is an external view of the camera body 1. FIG.

In FIG. 1A, the camera body 1 includes a photographing/detecting section 10, a battery section 90, and a connection section 80 that connects the photographing/detecting section 10 and the battery section 90 (power supply means).

The photographing/detecting unit 10 includes a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, an LED 17, and microphones 19L and 19R.

The face direction detection window 13 detects infrared rays emitted from an infrared LED lighting circuit 21 ( FIG. or its reflected light.

The start switch 14 is a switch for starting imaging.

A stop switch 15 is a switch for stopping imaging.

The imaging lens 16 guides the light rays to be imaged to the solid-state imaging element 42 (FIG. 5) inside the imaging/detecting section 10 .

The LED 17 is an LED that indicates that an image is being captured or indicates a warning.

The microphones 19R and 19L are microphones that take in surrounding sounds. The microphone 19L takes in sounds from the left side of the user's surroundings (the right side in FIG. 1A), and the microphone 19R takes in the sounds from the right side of the user's surroundings (the right side in FIG. 1A). to the left).

FIG. 1B is a diagram showing how the camera body 1 is hung by the user.

When the battery unit 90 is attached to the user's back side and the photographing/detecting unit 10 is attached to the front side of the user's body, the connection unit 80 connects both ends to the vicinity of the left and right ends of the photographing/detecting unit 10 . It is biased and supported in the chest direction. As a result, the photographing/detecting unit 10 is positioned in front of the user's clavicle. At this time, the face direction detection window 13 is located under the chin of the user. Within the face orientation detection window 13 is an infrared condensing lens 26 which is illustrated later in FIG. 2E. The optical axis of the imaging lens 16 (imaging optical axis) and the optical axis of the infrared condensing lens 26 (detection optical axis) are directed in different directions. The user's observation direction is detected from the position of . As a result, an imaging unit 40 (imaging means), which will be described later, can capture an image in the observation direction.

How to adjust the set position according to individual differences in body shape and clothes will be described later.

In addition, by arranging the photographing/detecting unit 10 on the front of the body and the battery unit 90 on the back, the weight is distributed, the fatigue of the user is reduced, and the displacement due to centrifugal force or the like when the user moves. has the effect of suppressing

In this embodiment, an example is shown in which the imaging/detecting unit 10 is attached so as to be positioned in front of the collarbone of the user, but the present invention is not limited to this. That is, if the camera body 1 can detect the user's observation direction by the face direction detection unit 20 and can capture an image in the observation direction by the imaging unit 40, the camera body 1 can be used on the user's body other than the head. may be attached to any of the

FIG. 1C is a view of the battery section 90 viewed from the rear of FIG. 1A.

In FIG. 1C, the battery section 90 includes a charging cable insertion port 91, adjustment buttons 92L and 92R, and a notch 93 for protecting the spine.

The charging cable insertion port 91 is an insertion port for a charging cable (not shown), through which the internal battery 94 is charged from an external power source and power is supplied to the photographing/detecting section 10 .

The adjustment buttons 92L and 92R are buttons for adjusting the length of the band portions 82L and 82R of the connecting portion 80. As shown in FIG. The adjustment button 92L is a button for adjusting the left band portion 82L, and the adjustment button 92R is a button for adjusting the right band portion 82R. In this embodiment, the lengths of the band portions 82L and 82R are adjusted independently by the adjustment buttons 92L and 92R, but the lengths of the band portions 82L and 82R are adjusted simultaneously by one button. good too. The band sections 82L and 82R are collectively referred to as a band section 82 below.

The spine avoidance notch 93 is a notch portion that avoids the spine portion so that the battery portion 90 does not come into contact with the spine portion of the user. By avoiding the convex part of the spine of the human body, it reduces the discomfort of wearing it, and at the same time prevents the body from moving left and right during use.

FIG. 1D is an external view of a display device 800 as a mobile device according to the first embodiment, which is configured separately from the camera body 1. FIG.

1D, the display device 800 includes a button A802, a display unit 803, a button B804, an in-camera 805, a face sensor 806, an angular velocity sensor 807, and an acceleration sensor 808. FIG. Also, although not shown in FIG. 1D, a wireless LAN capable of high-speed connection with the camera body 1 is provided.

The button A802 is a button having the function of the power button of the display device 800, and accepts a power ON/OFF operation by a long press, and accepts other processing timing instructions by a short press.

A display unit 803 can check an image captured by the camera body 1 and display a menu screen necessary for setting. In this embodiment, a transparent touch sensor is also provided on the upper surface of the display unit 803, and receives an operation by touching a screen being displayed (for example, a menu screen).

The button B804 is a button that functions as a calibration button 854 used for calibration processing, which will be described later.

The in-camera 805 is a camera capable of capturing an image of a person observing the display device 800 .

A face sensor 806 detects the face shape and viewing direction of a person viewing the display device 800 .
Although the specific structure of the face sensor 806 is not particularly limited, it can be implemented with various sensors such as a structured light sensor, a ToF sensor, and a millimeter wave radar.

Since the angular velocity sensor 807 is inside the display device 800, it is indicated by a dotted line as a perspective view. Since the display device 800 of the present embodiment also has a function of a calibrator, which will be described later, it is equipped with gyro sensors in the three-dimensional X, Y, and Z directions.

An acceleration sensor 808 detects the orientation of the display device 800 .

A general smart phone is used for the display device 800 according to the present embodiment, and the camera system according to the present invention can be implemented by making the firmware in the smart phone compatible with the firmware on the camera body 1 side. there is However, it is also possible to implement the camera system according to the present invention by adapting the firmware of the camera body 1 side to the application and OS of the smartphone as the display device 800 .

2A to 2F are diagrams illustrating the imaging/detection unit 10 in detail. In the subsequent figures, the same numbers are assigned to the parts that have already been explained, meaning the same functions, and the explanation in this specification is omitted.

FIG. 2A is a front view of the photographing/detecting unit 10. FIG.

The connection part 80 is composed of a right connection part 80R on the right side of the user's body (left side as viewed in FIG. 2A) and a left connection part 80L configured on the left side of the user's body (right side in FIG. 2A). It connects with the photographing/detecting unit 10 . Specifically, the connecting portion 80 is divided into an angle holding portion 81 made of a hard material and a band portion 82 for holding an angle with respect to the photographing/detecting portion 10 . That is, the right connecting portion 80R has an angle holding portion 81R and a band portion 82R, and the left connecting portion 80L has an angle holding portion 81L and a band portion 82L.

FIG. 2B is a diagram showing the shape of the band portion 82 of the connection portion 80. As shown in FIG. In this figure, the angle holding portion 81 is seen through in order to show the shape of the band portion 82 .

The band portion 82 comprises a connecting surface 83 and an electrical cable 84 .

The connection surface 83 is a connection surface between the angle holding portion 81 and the band portion 82, and has a cross-sectional shape that is not a perfect circle, here an elliptical shape. Hereinafter, among the connection surfaces 83, the connection surfaces 83 arranged bilaterally symmetrically on the right side (left side as viewed in FIG. 2B) and left side (right side as viewed in FIG. 2B) of the user's body when the camera body 1 is attached will be described. , the right connecting surface 83R and the left connecting surface 83L. The right connection surface 83R and the left connection surface 83L are shaped like the Japanese katakana character "C". That is, the distance between the right connection surface 83R and the left connection surface 83L becomes shorter as it goes upward from the bottom toward FIG. 2B. As a result, when the user hangs the camera body 1, the longitudinal direction of the connecting surface 83 of the connecting portion 80 is aligned with the user's body. There is an effect that the photographing/detecting unit 10 is comfortable and does not move in the left, right, front, and rear directions.

The electric cable 84 (power supply means) is a cable that is wired inside the band portion 82</b>L and electrically connects the battery portion 90 and the imaging/detecting portion 10 . The electric cable 84 connects the power source of the battery section 90 to the photographing/detecting section 10 and transmits/receives electric signals to/from the outside.

FIG. 2C is a diagram of the photographing/detecting unit 10 as seen from the back side. Since FIG. 2C is a view from the side in contact with the user's body, that is, the opposite side of FIG. 2A, the positional relationship between the right connecting portion 80R and the left connecting portion 80L is reversed from that in FIG. 2A.

The photographing/detecting section 10 has a power switch 11, an imaging mode switch 12, and a chest connection pad 18 on its back side.

The power switch 11 is a power switch for switching ON/OFF of the power of the camera body 1 . The power switch 11 of this embodiment is a switch in the form of a slide lever, but is not limited to this. For example, the power switch 11 may be a push-type switch, or may be a switch integrated with a slide cover (not shown) of the imaging lens 16 .

The imaging mode switch 12 (changer) is a switch for changing the imaging mode, and can change the mode related to imaging. In this embodiment, the imaging mode switch 12 can switch to a preset mode set using the display device 800, which will be described later, in addition to the still image mode and moving image mode. In this embodiment, the imaging mode switch 12 is a switch in the form of a slide lever that can select one of "Photo", "Normal", and "Pri" shown in FIG. 2C by sliding the lever. The imaging mode shifts to still image mode by sliding to "Photo", shifts to moving image mode by sliding to "Normal", and shifts to preset mode by sliding to "Pri". Note that the imaging mode switch 12 is not limited to the mode of this embodiment as long as it is a switch that can change the imaging mode.
For example, the imaging mode switch 12 may be composed of three buttons "Photo", "Normal", and "Pri".

The chest connection pad 18 (fixing means) is a portion that comes into contact with the user's body when the imaging/detection unit 10 is urged against the user's body. As shown in FIG. 2A, the photographing/detecting unit 10 is shaped so that its horizontal (left and right) length is longer than its vertical (up and down) length when worn. They are arranged near the left and right ends. By arranging in this way, it becomes possible to suppress left-right rotational blurring during imaging by the camera body 1 .
In addition, the presence of the chest connection pad 18 can prevent the power switch 11 and the imaging mode switch 12 from coming into contact with the body. Furthermore, the chest connection pad 18 serves to prevent the heat from being transmitted to the user's body even if the temperature of the imaging/detecting unit 10 rises due to long-time imaging, and to adjust the angle of the imaging/detecting unit 10. is also responsible.

FIG. 2D is a top view of the photographing/detecting unit 10. FIG.

As shown in FIG. 2D , a face direction detection window 13 is provided in the center of the upper surface of the photographing/detecting section 10 , and a chest connection pad 18 protrudes from the photographing/detecting section 10 .

FIG. 2E is a diagram showing the configuration of the infrared detection processing device 27 inside the imaging/detection unit 10 and arranged below the face direction detection window 13 .

The infrared detection processing device 27 includes an infrared LED 22 and an infrared condensing lens 26 .

The infrared LED 22 projects infrared rays 23 (FIG. 5) toward the user.

The infrared condensing lens 26 is a lens that forms an image of the reflected light beam 25 (FIG. 5) reflected from the user when the infrared ray 23 is projected from the infrared LED 22 on a sensor (not shown) of the infrared detection processing device 27. .

FIG. 2F is a view of the state in which the user hangs the camera body 1 as viewed from the left side of the user.

The angle adjustment button 85L is a button provided on the angle holding section 81L, and is used when adjusting the angle of the photographing/detecting section 10. FIG. Although not shown in this figure, an angle adjustment button 85R is also set inside the angle holding portion 81R on the opposite side at a position symmetrical to the angle adjustment button 85L. Hereinafter, when the angle adjustment buttons 85R and 85L are collectively referred to as the angle adjustment button 85. FIG.

The angle adjustment button 85 is also visible in FIGS. 2A, 2C, and 2D, but is omitted for simplicity of explanation.

The user can change the angle between the photographing/detecting unit 10 and the angle holding unit 81 by moving the angle holding unit 81 up and down toward FIG. 2F while pressing the angle adjustment button 85 . In addition, the chest connection pad 18 can change its projection angle. The photographing/detecting unit 10 adjusts the orientation of the imaging lens 16 horizontally to accommodate individual differences in the shape of the user's chest position by the action of these two angle changing members (the angle adjustment button 85 and the chest connection pad 18). It is possible to adjust.

FIG. 3 is a diagram illustrating the details of the battery section 90. As shown in FIG.

FIG. 3A is a partially see-through view of the battery section 90 from the back.

As shown in FIG. 3A, in order to balance the weight of the battery section 90, two batteries, a left battery 94L and a right battery 94R (hereinafter also collectively referred to as batteries 94), are symmetrically mounted inside. . By symmetrically arranging the battery 94 with respect to the central portion of the battery section 90 in this way, the left and right weight balance is adjusted and the positional deviation of the camera body 1 is prevented. Note that the battery section 90 may have a configuration in which only one battery is mounted.

FIG. 3(b) is a top view of the battery section 90. As shown in FIG. Also in this figure, the battery 94 is shown transparently.

As shown in FIG. 3(b), the relationship between the spine protection notch 93 and the battery 94 can be seen. In this way, by symmetrically arranging the batteries 94 on both sides of the notch 93 for protecting the spine, the relatively heavy battery section 90 can be worn by the user without burden.

FIG. 3C is a diagram of the battery section 90 viewed from the back side. FIG. 3(c) is a view seen from the side in contact with the user's body, that is, the opposite side of FIG. 3(a).

As shown in FIG. 3(c), a spine relief notch 93 is provided centrally along the user's spine.

FIG. 4 is a functional block diagram of the camera body 1. As shown in FIG. Since the details will be described later, the general flow of processing executed by the camera body 1 will be described here with reference to FIG.

4, the camera body 1 includes a face direction detection unit 20, a recording direction/angle of view determination unit 30, a photographing unit 40, an image clipping/development processing unit 50, a primary recording unit 60, a transmission unit 70, and other control unit 111. Prepare. These functional blocks are executed under the control of an overall control CPU 101 (FIG. 5) that performs overall control of the camera body 1. FIG.

The face direction detection unit 20 (face direction detection means) is a functional block executed by the infrared LED 22 and the infrared detection processing device 27 described above, and detects the face direction to analogize the observation direction. This is passed to the recording direction/angle of view determination unit 30 .

The recording direction/angle of view determining unit 30 (clipping region determining means) performs various calculations based on the observation direction inferred by the face direction detecting unit 20, and determines the position and range (clipping area) is determined, and this information is transferred to the image clipping/development processing unit 50 .

The photographing unit 40 (imaging means) converts the light beam from the object into an image, and passes the image to the image clipping/development processing unit 50 .

The image clipping/development processing unit 50 uses the information from the recording direction/angle of view determination unit 30 to clip and develop the image from the photographing unit 40, so that only the image in the direction in which the user is looking is primary. It is passed to the recording unit 60 .

The primary recording unit 60 is a functional block configured by the primary memory 103 (FIG. 5) and the like, records video information, and transfers it to the transmitting unit 70 at the necessary timing.

The transmission unit 70 wirelessly connects with the display device 800 (FIG. 1D), the calibrator 850, and the simple display device 900, which are predetermined communication partners, and communicates with them.

The display device 800 is a display device that can be connected to the transmission unit 70 via a wireless LAN (hereinafter referred to as "high-speed wireless") that allows high-speed connection. Here, in this embodiment, wireless communication corresponding to the IEEE802.11ax (WiFi 6) standard is used for high-speed wireless communication, but wireless communication corresponding to other standards such as the WiFi 4 standard and the WiFi 5 standard is used. good too. Further, the display device 800 may be a device developed exclusively for the camera body 1, or may be a general smart phone, a tablet terminal, or the like.

The transmission unit 70 and the display device 800 may be connected using low-power radio, or may be connected by both high-speed radio and low-power radio, or may be connected by switching. In this embodiment, a large amount of data, such as a video file of a moving image, which will be described later, is transmitted by high-speed wireless transmission, and light data or data that may take a long time to transmit is transmitted by low-power wireless transmission. In this embodiment, Bluetooth is used for low-power wireless communication, but other short-range wireless communication such as NFC (Near Field Communication) may be used.

The calibrator 850 is a device that performs initial settings of the camera body 1 and individual settings, and is a device that can be connected to the transmission unit 70 wirelessly at high speed, like the display device 800 . Details of the calibrator 850 will be described later. Also, the display device 800 may also function as the calibrator 850 .

The simple display device 900 is a display device that can be connected to the transmitter 70 only by low-power radio, for example.

The simple display device 900 is a display device that, although it cannot transmit a moving image with the transmission unit 70 due to time constraints, can transmit the timing of imaging start/stop and can confirm images such as composition confirmation. As with the display device 800, the simple display device 900 may be a device developed exclusively for the camera body 1, or may be a smart watch or the like.

FIG. 5 is a block diagram showing the hardware configuration of the camera body 1. As shown in FIG. Further, the configurations and functions described with reference to FIGS. 1A to 1C and the like are denoted by the same reference numerals, and detailed description thereof will be omitted.

5, the camera body 1 includes an overall control CPU 101, a power switch 11, an imaging mode switch 12, a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, and an LED 17.

The camera body 1 also includes an infrared LED lighting circuit 21, an infrared LED 22, an infrared condensing lens 26, and an infrared detection processing device 27, which constitute the face direction detection section 20 (FIG. 4).

The camera body 1 also includes an imaging unit 40 (FIG. 4) comprising an imaging driver 41, a solid-state imaging device 42, and an imaging signal processing circuit 43, and a transmitting unit 70 (FIG. 4) comprising a low-power wireless unit 71 and a high-speed wireless unit 72. 4).

Although only one imaging unit 40 is provided in the present embodiment, the camera body 1 is provided with two or more imaging units 40 to capture a 3D image or an image that can be acquired by one imaging unit 40. A wide-angle image may be captured, or images may be captured in a plurality of directions.

The camera body 1 also includes various memories such as a large-capacity nonvolatile memory 51 , an internal nonvolatile memory 102 , and a primary memory 103 .

Further, the camera body 1 includes an audio processing section 104 , a speaker 105 , a vibrating body 106 , an angular velocity sensor 107 , an acceleration sensor 108 and various switches 110 .

The overall control CPU 101 is connected to the power switch 11 described above with reference to FIG. 2C, and controls the camera body 1 . The recording direction/angle of view determination unit 30, image clipping/development processing unit 50, and other control unit 111 shown in FIG. 4 are configured by the overall control CPU 101 itself.

The infrared LED lighting circuit 21 controls lighting and extinguishing of the infrared LED 22 described above with reference to FIG. 2E, and controls projection of infrared rays 23 from the infrared LED 22 toward the user.

The face direction detection window 13 is composed of a visible light cut filter, which hardly transmits visible light, but sufficiently transmits infrared ray 23 which is light in the infrared region and its reflected light 25 .

The infrared condensing lens 26 is a lens that condenses the reflected light beam 25 .

The infrared detection processing device 27 has a sensor that detects the reflected light beam 25 condensed by the infrared condensing lens 26 . This sensor forms an image of the collected reflected light beam 25 , converts it into sensor data, and transfers it to the overall control CPU 101 .

When the user hangs the camera body 1 as shown in FIG. 1B, the face direction detection window 13 is located under the user's chin. Therefore, the infrared rays 23 projected from the infrared LED lighting circuit 21 pass through the face direction detection window 13 and are irradiated onto the infrared irradiation surface 24 near the chin of the user as shown in FIG. Also, the infrared rays 23 reflected by the infrared irradiation surface 24 become reflected light rays 25 , pass through the face direction detection window 13 , and are condensed by the infrared condensing lens 26 to the sensor in the infrared detection processing device 27 .

The various switches 110 are not shown in FIGS. 1A to 1C and the like, and although the details are omitted, they are switches for executing functions unrelated to this embodiment.

The imaging driver 41 includes a timing generator and the like, and generates and outputs various timing signals to each unit related to imaging to drive imaging.

The solid-state imaging device 42 outputs a signal obtained by photoelectrically converting the subject image projected from the imaging lens 16 described with reference to FIG. 1A to the imaging signal processing circuit 43 .

The imaging signal processing circuit 43 outputs imaging data generated by performing processing such as clamping and processing such as A/D conversion on the signal from the solid-state imaging device 42 to the overall control CPU 101 .

The built-in non-volatile memory 102 is a flash memory or the like, and stores a startup program for the overall control CPU 101 and setting values for various program modes. In the present embodiment, since it is possible to change the observation field (angle of view) and set the effect level of anti-vibration control, such set values are also recorded.

The primary memory 103 is composed of a RAM or the like, and temporarily stores video data being processed and temporarily stores the calculation results of the overall control CPU 101 .

A large-capacity nonvolatile memory 51 records or reads primary image data. In this embodiment, for simplification of explanation, a case where the large-capacity nonvolatile memory 51 is a semiconductor memory without a detachable mechanism will be explained, but the present invention is not limited to this. For example, the large-capacity nonvolatile memory 51 may be composed of a removable recording medium such as an SD card, or may be used together with the built-in nonvolatile memory 102 .

The low-power radio unit 71 exchanges data with the display device 800, the calibrator 850, and the simple display device 900 by low-power radio.

The high-speed radio unit 72 exchanges data with the display device 800, the calibrator 850, and the simple display device 900 by high-speed radio.

The audio processing unit 104 includes a microphone 19L on the right side in FIG. 1A and a microphone 19R on the left side in FIG. 1A for collecting external sounds (analog signals), and processes the collected analog signals to generate audio signals do.

The LED 17, the speaker 105, and the vibrator 106 emit light, emit sound, or vibrate to notify or warn the user of the state of the camera body 1. FIG.

The angular velocity sensor 107 is a sensor using a gyro or the like, and detects movement of the camera body 1 itself as gyro data.

The acceleration sensor 108 detects the orientation of the imaging/detecting unit 10 .

The angular velocity sensor 107 and the acceleration sensor 108 are built in the photographing/detecting unit 10, and the angular velocity sensor 807 and the acceleration sensor 808 are separately provided in the display device 800 described later. .

FIG. 6 is a block diagram showing the hardware configuration of the display device 800. As shown in FIG. For simplification of explanation, the parts explained with reference to FIG. 1D are denoted by the same reference numerals and the explanation thereof is omitted.

6, the display device 800 includes a display device control unit 801, a button A 802, a display unit 803, a button B 804, an in-camera 805, a face sensor 806, an angular velocity sensor 807, an acceleration sensor 808, an imaging signal processing circuit 809, and various switches. 811.

The display device 800 also includes a built-in non-volatile memory 812, a primary memory 813, a large-capacity non-volatile memory 814, a speaker 815, a vibrating body 816, an LED 817, an audio processing section 820, a low-power radio unit 871, and a high-speed radio unit 872. Prepare.

The display device control unit 801 is configured by a CPU, is connected to the button A 802 described using FIG. 1D, the face sensor 806, and the like, and controls the display device 800.

The imaging signal processing circuit 809 has functions equivalent to those of the imaging driver 41, the solid-state imaging device 42, and the imaging signal processing circuit 43 inside the camera body 1, but is not so important in the description of this embodiment, so the description is simplified. Therefore, they are explained together. Data output from the imaging signal processing circuit 809 is processed within the display device control unit 801 . Details of processing of this data will be described later.

Various switches 811 are not shown in FIG. 1D and are switches for executing functions unrelated to this embodiment, although the details are omitted.

An angular velocity sensor 807 is a sensor using a gyro or the like, and detects movement of the display device 800 itself.

An acceleration sensor 808 detects the orientation of the display device 800 itself.

As described above, the angular velocity sensor 807 and the acceleration sensor 808 are built in the display device 800 and have the same functions as the angular velocity sensor 107 and the acceleration sensor 808 in the camera body 1 described above, but are separate components. is.

A built-in non-volatile memory 812 uses a flash memory or the like, and stores a startup program for the display device control unit 801 and setting values for various program modes.

A primary memory 813 is configured by a RAM or the like, and temporarily stores video data being processed and temporarily stores the calculation results of the imaging signal processing circuit 809 . In this embodiment, during video recording, the gyro data detected by the angular velocity sensor 107 at the imaging time of each frame is associated with each frame and held in the primary memory 813 .

A large-capacity nonvolatile memory 814 records or reads image data of the display device 800 .
In this embodiment, the large-capacity non-volatile memory 814 is composed of removable memory such as an SD card. A non-detachable memory such as the large-capacity non-volatile memory 51 in the camera body 1 may be used.

The speaker 815, vibrator 816, and LED 817 emit sound, vibrate, or emit light to notify or warn the user of the state of the display device 800. FIG.

The audio processing unit 820 includes a left microphone 819L and a right microphone 819R that collect external sounds (analog signals), processes the collected analog signals, and generates audio signals.

The low-power radio unit 871 exchanges data with the camera body 1 by low-power radio.

The high-speed radio unit 872 exchanges data with the camera body 1 by high-speed radio.

Face sensor 806 includes infrared LED lighting circuit 821 , infrared LED 822 , infrared condensing lens 826 , and infrared detection processing device 827 .

The infrared LED lighting circuit 821 is a circuit having the same function as the infrared LED lighting circuit 21 in FIG. to control the light emission of the

The infrared condensing lens 826 is a lens that condenses the reflected light beam 825 of the infrared rays 823 .

Infrared detection processing device 827 has a sensor that detects the reflected light beams collected by infrared collecting lens 826 . This sensor converts the collected reflected light beam 825 into sensor data and passes it to the display device controller 801 .

When the face sensor 806 shown in FIG. 1D is directed toward the user, as shown in FIG. 6, an infrared ray 823 projected from an infrared LED lighting circuit 821 is irradiated onto an infrared irradiation surface 824 that is the entire face of the user. be. Also, the infrared rays 823 reflected by the infrared irradiation surface 824 become reflected rays 825 and are condensed by the infrared condensing lens 826 to the sensor in the infrared detection processing device 827 .

Although details are omitted, the other function unit 830 is a function that is not related to the present embodiment, and executes functions unique to the smartphone such as a telephone function and other sensors.

How to use the camera body 1 and the display device 800 will be described below.

FIG. 7A is a flow chart showing an outline of image recording processing according to the present embodiment, which is executed in the camera body 1 and the display device 800. FIG.

As an aid to the explanation, in FIG. 7A, which device shown in FIG. 4 is performing the step is described on the right side of each step. That is, steps S100 to S700 in FIG. 7A are executed by the camera body 1, and steps S800 to S1000 in FIG. 7A are executed by the display device 800. FIG.

In step S100, when the power switch 11 is turned on and the camera body 1 is powered on, the overall control CPU 101 is activated and reads out the activation program from the built-in non-volatile memory 102. FIG. After that, the overall control CPU 101 executes preparatory operation processing for setting the camera body 1 before imaging. Details of the preparatory operation process will be described later with reference to FIG. 7B.

In step S200, the face direction detection unit 20 detects the face direction, and performs face direction detection processing for analogizing the viewing direction. The details of the face direction detection process will be described later with reference to FIG. 7C. This process is executed at a predetermined frame rate.

In step S300, the recording direction/angle of view determination unit 30 executes recording direction/range determination processing. Details of the recording direction/range determination process will be described later with reference to FIG. 7D.

In step S400, the imaging unit 40 performs imaging and generates imaging data.

In step S500, the image clipping/development processing unit 50 clips an image from the imaging data generated in step S400 using the recording direction and angle of view information determined in step S300, and performs development processing of that range. Execute recording area development processing. Details of the recording area development processing will be described later with reference to FIG. 7E.

In step S600, the primary recording unit 60 executes primary recording processing in which the image developed in step S500 is stored in the primary memory 103 as image data. Details of the primary recording process will be described later with reference to FIG.

In step S700, the transmission unit 70 executes a transfer process to the display device 800 for wirelessly transmitting the video primarily recorded in step S600 to the display device 800 at a designated timing. Details of the transfer processing to the display device 800 will be described later with reference to FIG. 16 .

The steps after step S<b>800 are executed by display device 800 .

In step S800, the display device control unit 801 executes optical correction processing for optically correcting the image transferred from the camera body 1 in step S700. Details of the optical correction processing will be described later with reference to FIG.

In step S900, the display device control unit 801 performs image stabilizing processing on the image optically corrected in step S800. Details of the anti-vibration processing will be described later with reference to FIG. 19 .

Note that the order of steps S800 and S900 may be reversed. In other words, image stabilization correction may be performed first, and then optical correction may be performed.

In step S1000, the display device control unit 801 performs secondary recording to record the image for which the optical correction processing and image stabilization processing in steps S800 and S900 have been completed in the large-capacity nonvolatile memory 814, and ends this processing.

Next, with reference to FIGS. 7B to 7F, the subroutine of each step described with reference to FIG. 7A will be described in detail along with the order of processing and other figures.

FIG. 7B is a flowchart of a subroutine of the preparatory operation process in step S100 of FIG. 7A. This process will be described below using the parts shown in FIGS. 2 and 5. FIG.

In step S101, it is determined whether the power switch 11 is ON. If the power remains OFF, it waits, and if it turns ON, the process proceeds to step S102.

In step S102, the mode selected by the imaging mode switch 12 is determined.
As a result of determination, if the mode selected by the imaging mode switch 12 is the moving image mode, the process proceeds to step S103.

In step S103, various settings of the moving image mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, after which the process proceeds to step S104. Here, the various settings of the moving image mode include the angle of view setting value ang (which is preset to 90° in this embodiment) and the image stabilizing level designated as "strong", "medium", "off", and the like.

In step S104, after starting the operation of the imaging driver 41 for moving image mode, this subroutine is exited.

If the result of determination in step S102 is that the mode selected by the imaging mode switch 12 is the still image mode, the process proceeds to step S106.

In step S106, various settings of the still image mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, after which the process proceeds to step S107. Here, the various settings of the still image mode include the angle of view setting value ang (preset to 45° in this embodiment) and the image stabilization level specified by "strong", "medium", "off", etc. .

In step S107, after starting the operation of the imaging driver 41 for the still image mode, this subroutine is exited.

If the result of determination in step S102 is that the mode selected by the imaging mode switch 12 is the preset mode, the process proceeds to step S108. Here, the preset mode is a mode in which an imaging mode is set for the camera body 1 from an external device such as the display device 800, and is one of three imaging modes that can be switched by the imaging mode switch 12. .
The preset mode is a mode for custom shooting. Since the camera body 1 is a small wearable device, the camera body 1 is not provided with operation switches or setting screens for changing detailed settings. The detailed settings of the main body 1 are changed.

For example, consider a case where it is desired to continuously capture the same moving image at a field angle of 90° and a field angle of 110°. Since the angle of view is set to 90° in the normal moving image mode, when performing such image capturing, first, after image capturing in the normal moving image mode, the moving image capturing is finished once, and the display device 800 is moved to the camera body 1. It is necessary to switch to the setting screen and switch the angle of view to 110 degrees.
However, such an operation on the display device 800 is troublesome during some event.

On the other hand, if the preset mode is set in advance to a mode for shooting a movie with an angle of view of 110°, after shooting a movie with an angle of view of 90°, all you have to do is slide the shooting mode switch 12 to "Pri". , the angle of view of 110° can be changed immediately. That is, the user does not need to interrupt the current action and perform the above-described troublesome operations.

In addition, the settings in the preset mode include not only the angle of view, but also the image stabilization level specified by "strong", "medium", "off", etc., and voice recognition settings that are not explained in this embodiment. good too.

In step S108, various settings of the preset mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, after which the process proceeds to step S109. Here, the various settings in the preset mode include the angle of view setting value ang and the image stabilizing level designated by "strong", "medium", "off", and the like.

In step S109, after starting the operation of the imaging driver 41 for the preset mode, this subroutine is exited.

Here, various settings of the moving image mode read in step S103 will be described with reference to FIG.

FIG. 13 is a diagram showing a menu screen for various settings of the moving image mode, which is displayed on the display unit 803 of the display device 800 before imaging by the camera body 1. As shown in FIG. Note that the same reference numerals are used for the same portions as those in FIG. 1D, and the description thereof is omitted. Note that the display portion 803 has a touch panel function, and the following description will be made on the assumption that it functions by a touch operation including an operation such as a swipe.

13, the menu screen includes a preview screen 831, a zoom lever 832, a recording start/stop button 833, a switch 834, a remaining battery level display 835, a button 836, a lever 837, and an icon display section 838.

On the preview screen 831, it is possible to confirm the image captured by the camera body 1, and it is possible to confirm the zoom amount and the angle of view.

A zoom lever 832 is an operation unit that allows zoom setting by shifting left and right. In this embodiment, four values of 45°, 90°, 110°, and 130° can be set as the angle of view setting value ang. You may make it possible.

The recording start/stop button 833 is a toggle switch having both the functions of the start switch 14 and the stop switch 15 .

A switch 834 is a switch for switching between "OFF" and "ON" of anti-vibration.

The remaining battery level display 835 displays the remaining battery level of the camera body 1 .

A button 836 is a button for entering other modes.

A lever 837 is a lever for setting the vibration isolation strength. In this embodiment, only "strong" and "medium" can be set as vibration isolation strength, but other vibration isolation strengths such as "weak" may also be set. Also, the anti-vibration intensity may be set steplessly.

The icon display portion 838 displays a plurality of thumbnail icons for preview.

FIG. 7C is a flowchart of a subroutine of face direction detection processing in step S200 of FIG. 7A. Before explaining the details of this process, a face direction detection method using infrared light projection will be explained with reference to FIGS. 8A to 8K.

FIG. 8A is a diagram showing an image of the user seen through the face direction detection window 13. FIG.

The image of FIG. 8A is obtained when the face direction detection window 13 does not have a visible light cut filter component, the visible light is sufficiently transmitted, and the infrared detection processing device 27 is a visible light imaging device. It is the same as the image captured by the device.

The image of FIG. 8A shows the user's face 204 including the neck front 201 above the collarbone, the base of the chin 202, the tip of the chin 203, and the nose.

FIG. 8B is a diagram showing a case where a fluorescent lamp in the room is reflected as a background in the image of the user seen through the face direction detection window 13. In FIG.

The image of FIG. 8B shows a plurality of fluorescent lights 205 around the user. As described above, various backgrounds and the like are reflected in the infrared detection processing device 27 depending on the conditions of use. difficult to separate. Recently, there is a technique for segmenting such images by using AI or the like, but it requires a high capability of the overall control CPU 101 and is not suitable for the camera body 1, which is a portable device.

Actually, since the face direction detection window 13 is composed of a visible light cut filter, almost no visible light is transmitted. Therefore, the image of the infrared detection processing device 27 does not become the image shown in FIGS. 8A and 8B. .

FIG. 8C shows the case where the user shown in FIG. 8B and the fluorescent lamp as the background thereof are imaged by the sensor of the infrared detection processing device 27 through the face direction detection window 13 in a state in which the infrared LED 22 is not turned on. is a diagram showing an image of .

In the image of FIG. 8C, the user's neck and chin are darkened. On the other hand, the fluorescent lamp 205 appears rather bright because it has not only visible rays but also infrared rays.

In FIG. 8D, the user shown in FIG. 8B and the fluorescent lamp as the background thereof are imaged by the sensor of the infrared detection processing device 27 through the face direction detection window 13 with the infrared LED 22 turned on. It is a figure which shows the image|video in a case.

In the image of FIG. 8D, the user's neck and chin are brightened. On the other hand, unlike FIG. 8C, the brightness around the fluorescent lamp 205 does not change.

FIG. 8E is a diagram showing a difference image calculated from the images of FIGS. 8C and 8D. It can be seen that the user's face is highlighted.

In this manner, the overall control CPU 101 extracts the user's face by calculating the difference between the images formed by the sensor of the infrared detection processing device 27 when the infrared LED 22 is lit and when the infrared LED 22 is extinguished. A difference image (hereinafter also referred to as a face image) is obtained.

The face direction detection unit 20 of this embodiment employs a method of obtaining a face image by extracting the infrared reflection intensity as a two-dimensional image by the infrared detection processing device 27 . The sensor of the infrared detection processing device 27 has a structure similar to that of a general imaging device, and acquires a face image frame by frame. A vertical synchronizing signal (hereinafter referred to as a V signal) for synchronizing the frame is generated by the infrared detection processor 27 and output to the overall control CPU 101 .

FIG. 9 is a timing chart showing the timing of turning on/off the infrared LED 22. As shown in FIG.

FIG. 9(a) shows the timing at which the V signal is generated by the infrared detection processor 27. FIG. When the V signal becomes Hi, the timing of frame synchronization and turning on/off of the infrared LED 22 is measured.

In FIG. 9A, t1 indicates the first facial image acquisition period, and t2 indicates the second facial image acquisition period. FIGS. 9A, 9B, 9C, and 9D are drawn so that the time axes of the horizontal axes are the same.

In FIG. 9B, the vertical axis represents the H position of the image signal output from the sensor of the infrared detection processing device 27 . The infrared detection processor 27 controls the movement of the sensor so that the H position of the image signal is synchronized with the V signal, as shown in FIG. 9(b). As described above, the sensor of the infrared detection processing device 27 employs a structure similar to that of a general imaging device, and its movement is known, so detailed control will be omitted.

FIG. 9(c) shows the switching timing of the IR-ON signal output from the overall control CPU 101 to the infrared LED lighting circuit 21 between Hi and Low. The switching of the IR-ON signal between Hi and Low is controlled by the general control CPU 101 so as to synchronize with the V signal as shown in FIG. 9(c). Specifically, the overall control CPU 101 outputs a Low IR-ON signal to the infrared LED lighting circuit 21 during the period t1, and outputs a High IR-ON signal to the infrared LED lighting circuit 21 during the period t2. Output to circuit 21 .

Here, while the IR-ON signal is Hi, the infrared LED lighting circuit 21 lights the infrared LED 22, and the infrared ray 23 is projected to the user. On the other hand, while the IR-ON signal is Low, the infrared LED lighting circuit 21 turns off the infrared LED 22 .

FIG. 9(d) shows imaging data output from the sensor of the infrared detection processing device 27 to the overall control CPU 101. FIG. The vertical direction indicates the signal intensity, which indicates the amount of light received by the reflected light beam 25 . In other words, during the period t1, the infrared LED 22 is turned off, so there is no reflected light 25 from the user's face, and image data such as that shown in FIG. 8C is obtained. On the other hand, during the period t2, since the infrared LED 22 is lit, there is a reflected light beam 25 from the user's face, and imaging data as shown in FIG. 8D is obtained. Therefore, as shown in FIG. 9(d), the signal intensity during the period t2 is higher than the signal intensity during the period t1 by the reflected light beam 25 from the user's face.

FIG. 9(e) is obtained by taking the difference of the imaging data during each period of t1 and t2 in FIG. 9(d). Only the extracted imaging data is obtained.

FIG. 7C shows face direction detection processing in step S200 including the operations described above with reference to FIGS. 8C to 8E and FIG.

First, in step S201, when the V signal output from the infrared detection processing device 27 reaches the timing V1 at which the period t1 starts, the process proceeds to step S202.

Then, in step S202, the IR-ON signal is set to Low and output to the infrared LED lighting circuit 21. FIG. Thereby, the infrared LED 22 is extinguished.

In step S203, one frame of imaging data output from the infrared detection processing device 27 during the period t1 is read, and the data is temporarily stored in the primary memory 103 as Frame1.

In step S204, when the V signal output from the infrared detection processing device 27 reaches the timing V2 when the period t2 starts, the process proceeds to step S203.

At step S 205 , the IR-ON signal is set to Hi and output to the infrared LED lighting circuit 21 . Thereby, the infrared LED 22 is lit.

In step S206, one frame of imaging data output from the infrared detection processing device 27 during the period t2 is read, and the data is temporarily stored in the primary memory 103 as Frame2.

At step S 207 , the IR-ON signal is set to Low and output to the infrared LED lighting circuit 21 . As a result, the infrared LED 22 is extinguished.

In step S208, Frame1 and Frame2 are read out from the primary memory 103, and the difference obtained by subtracting Frame1 from Frame2 is calculated as the light intensity Fn of the 25 components of the reflected light beam of the user in FIG. corresponds to a process called black subtraction).

In step S209, the neck position (neck rotation center) is extracted from the light intensity Fn.

First, the overall control CPU 101 divides the face image into a plurality of distance areas based on the light intensity Fn, which will be explained using FIG. 8F.

FIG. 8F shows the distribution of the amount of light for each part of the user's face and neck. FIG. 10 is a diagram showing a case where the scale is adjusted to match with .

FIG. 8F(a) is a diagram showing the distribution of the light intensity of the reflected light ray 25 in the face image of FIG. 8E divided into regions and shown in gray stages for simplification of explanation. For the sake of explanation, the Xf axis is taken in the direction from the center of the user's neck to the tip of the chin.

In FIG. 8F(a), the horizontal axis indicates the light intensity on the Xf axis in FIG. 8F(a), and the vertical axis indicates the Xf axis. The horizontal axis indicates higher light intensity toward the right.

In FIG. 8F(a), the face image is divided into six areas (distance areas) 211 to 216 according to the light intensity.

Region 211 is the region with the highest light intensity and is shown in white as shades of gray.

Region 212 is a region of slightly less light intensity than region 211 and is shown in a much lighter gray color as a gray scale.

Region 213 is a region of even lower light intensity than region 212 and is shown in light gray as a gray scale.

Region 214 is a region of even lower light intensity than region 213 and is shown in intermediate gray as a gray scale.

Area 215 is an area where the light intensity is even lower than that of area 214, and is shown in slightly darker gray as a gray scale.

Area 216 is the area with the lowest light intensity and is the darkest shade of gray. The area above the area 216 is black with no light intensity.

This light intensity will be explained in detail below with reference to FIG.

FIG. 10 is a diagram for explaining vertical movement of the user's face, and shows a state observed from the left lateral direction of the user.

FIG. 10(a) is a diagram showing a state in which the user faces the front. An imaging/detecting unit 10 is located in front of the user's clavicle. Infrared rays 23 from an infrared LED 22 are emitted to the lower portion of the user's head from the face direction detection window 13 in the upper portion of the photographing/detecting section 10 . Dn is the distance from the face direction detection window 13 to the base of the neck 200 on the collarbone of the user, Db is the distance from the face direction detection window 13 to the base of the chin 202, and Db is the distance from the face direction detection window 13 to the chin 203. is Dc, the distance increases in the order of Dn, Db, and Dc. Since the light intensity is inversely proportional to the square of the distance, the light intensity when the reflected light beam 25 from the infrared irradiation surface 24 is imaged on the sensor of the infrared detection processing device 27 is 200 at the base of the neck and 202 at the base of the chin. , the chin 203 becomes weaker. Further, it can be seen that the light intensity of the face 204 including the nose, which is farther from the face direction detection window 13 than Dc, is even darker. That is, in the case shown in FIG. 10A, an image having the light intensity distribution shown in FIG. 8F is obtained.

The configuration of the face direction detection unit 20 is not limited to the configuration shown in this embodiment as long as the face direction of the user can be detected. For example, an infrared pattern may be emitted from the infrared LED 22 and the infrared pattern reflected from the irradiation target may be detected by the sensor of the infrared detection processing device 27 . In this case, the sensor of the infrared detection processing device 27 is preferably a structured light sensor. Further, the sensor of the infrared detection processing device 27 may be a sensor that performs phase comparison between the infrared rays 23 and the reflected light beam 25, such as a Tof sensor.

Next, extraction of the neck position in step S209 of FIG. 7C will be described with reference to FIG. 8G.

FIG. 8G (a) is a diagram in which the symbols indicating each part of the user's body in FIG. be.

A white region 211 corresponds to the base of the neck 200 (FIG. 10(a)), and a fairly light gray region 212 corresponds to the front neck 201 (FIG. 10(a)) and is bright. A gray area 213 corresponds to the base of the chin 202 (FIG. 10(a)). A middle gray area 214 corresponds to the tip of the chin 203 (FIG. 10(a)), and a slightly darker gray area 215 is located below the face 204 (FIG. 10(a)). It corresponds to the lower part of the face around the lips and the periphery. In addition, a darker gray area 216 corresponds to the nose located in the center of the face 204 (FIG. 10(a)) and the surrounding upper part of the face.

As shown in FIG. 10A, the distances Db and Dc are less different than the distances from the face direction detection window 13 to other parts of the user. There is also little difference in reflected light intensity in the gray area 214 of .

On the other hand, as shown in FIG. 10A, among the distances from the face direction detection window 13 to each part of the user, the distance Dn is the shortest distance. A region 211 is a portion having the highest reflection intensity.

Therefore, the overall control CPU 101 selects a position 206 indicated by a double circle in FIG. The neck rotation center position (hereinafter referred to as neck position 206) is set. The processing up to this point is the content performed in step S209 of FIG. 7C.

Next, extraction of the tip of the chin position in step S210 of FIG. 7C will be described with reference to FIG. 8G.

A medium gray area 214 that is lighter than the area 215 corresponding to the lower part of the face including the lips in the face 204 shown in FIG. 8G(a) is the area including the tip of the chin. As can be seen from FIG. 8G(a), the light intensity drops sharply in a region 215 in contact with the region 214, and the change in distance from the face direction detection window 13 increases. The overall control CPU 101 determines that the region 214 in front of the region 215 where the light intensity sharply drops is the chin region. Furthermore, the overall control CPU 101 calculates (extracts) the position 207 of the tip of the chin at the center of the area 214 in the left and right direction and farthest from the neck position 206 (the position indicated by the black circle in FIG. 8G(a)).

For example, FIGS. 8H and 8I show changes when the face is turned to the right.

FIG. 8H is a diagram showing a difference image calculated in the same manner as in FIG. 8E when the user's face is facing right. FIG. 8I is a diagram in which double circles and black circles indicating a neck position 206 and a chin position 207r, which are central positions of neck movement, are superimposed on FIG. 8H.

Since the user has turned to the right, area 214 moves to area 214r shown in FIG. A region 215 corresponding to the lower part of the face including the lips in the face 204 also moves to a leftward region 215r when viewed from the imaging/detecting unit 10 side.

Therefore, the overall control CPU 101 determines that the area 214r in front of 215r where the light intensity drops sharply is the chin area. Furthermore, the overall control CPU 101 calculates (extracts) the position of the left and right center of 214r and the farthest position from the neck position 206 (the position indicated by the black circle in FIG. 8I) as the chin position 207r.

After that, the overall control CPU 101 obtains a movement angle θr indicating how much the chin position 207r in FIG. 8I has moved from the chin position 207 in FIG. 8G(a) to the right around the neck position 206. As shown in FIG. 8I, the movement angle θr is the angle of the user's face in the horizontal direction.

By the above method, in step S210, the infrared detection processing device 27 of the face direction detection unit 20 (three-dimensional detection sensor) detects the chin position and the horizontal angle of the user's face.

Next, detection of the upward direction of the face will be described.

FIG. 10(b) is a diagram showing a state in which the user faces horizontally, and FIG. 10(c) is a diagram showing a state in which the user faces upward 33° from the horizontal direction. is.

In FIG. 10B, the distance from the face direction detection window 13 to the chin 203 is Ffh, and in FIG. 10C, the distance from the face direction detection window 13 to the chin 203u is Ffu.

As shown in FIG. 10(c), since the chin 203u moves upward together with the face, it can be seen that Ffu is longer than Ffh.

FIG. 8J is a diagram showing an image of the user seen through the face direction detection window 13 when the user faces 33° above the horizontal. As shown in FIG. 10(c), since the user is facing upward, the face 204 including lips and nose cannot be seen from the face direction detection window 13 located under the user's chin. I can see up to 203 ahead. FIG. 8K shows the light intensity distribution of the reflected light beam 25 when the user is irradiated with the infrared rays 23 at this time. FIG. 8K is a diagram in which double circles and black circles indicating the neck position 206 and the chin position 207u are superimposed on the difference image calculated by the same method as in FIG. 8E.

The six regions 211u to 216u corresponding to the light intensities in FIG. 8K are regions with the same light intensity as the regions shown in FIG. 8F, with “u” added. It can be seen that the light intensity of the user's chin 203 was in the middle gray area 214 in FIG. 8F, but shifted to the gray side in FIG. 8K and is in a slightly darker gray area 215u. Thus, as shown in FIG. 10(c), as a result of Ffu being longer than Ffh, the light intensity of the reflected light beam 25 from the tip of the user's chin 203 is weakened in inverse proportion to the square of the distance. , can be detected by the infrared detection processor 27 .

Next, detection of the downward direction of the face will be described.

FIG. 10(d) is a diagram showing a state in which the face of the user is directed downward by 22° from the horizontal direction.

In FIG. 10D, the distance from the face direction detection window 13 to the tip of the chin 203d is Ffd.

As shown in FIG. 10(d), since the chin 203d moves downward together with the face, Ffd is shorter than Ffh, and the light intensity of the reflected light ray 25 from the chin 203 increases.

Returning to FIG. 7C, in step S211, the overall control CPU 101 detects the face direction from the chin position based on the light intensity at the chin position detected by the infrared detection processing device 27 of the face direction detection unit 20 (three-dimensional detection sensor). A distance to the detection window 13 is calculated. Based on this, the vertical angle of the face is also calculated.

In step S212, the angle of the left-right direction (first detection direction) of the face obtained in steps S210 and S211 and the vertical direction (second detection direction) perpendicular thereto are converted into three-dimensional observation directions of the user. vi in primary memory 103 (where i is an arbitrary sign). For example, the observation direction vo when the user is observing the center of the front is 0° in the horizontal direction θh and 0° in the vertical direction θv, so the vector information is [0°, 0°]. Also, the observation direction vr when the user is observing 45 degrees to the right becomes vector information of [45 degrees, 0 degrees].

Although the vertical angle of the face is calculated by detecting the distance from the face direction detection window 13 in step S211, the method is not limited to this method. For example, the angle change may be calculated by comparing the variation level of the light intensity of the chin 203 . In other words, the gradient change of the reflected light intensity gradient CDu from the jaw base 202 to the chin 203 in FIG. Alternatively, the angle change of the jaw may be calculated.

FIG. 7D is a flowchart of a subroutine of the recording direction/recording range determination process in step S300 of FIG. 7A. Before describing the details of this process, first, using FIG. 11A, a super-wide-angle image for which the recording direction and recording range in this embodiment are determined will be described.

In the camera body 1 of the present embodiment, the photographing unit 40 uses the ultra-wide-angle imaging lens 16 to capture a super-wide-angle image around the photographing/detecting unit 10, and a part of the image is cut out to obtain an image in the observation direction. have achieved

FIG. 11A is a diagram showing the target field of view 125 in the ultra-wide-angle image captured by the imaging unit 40 when the user faces the front.

As shown in FIG. 11A, the imageable pixel area 121 of the solid-state imaging device 42 is a rectangular area. Also, the effective projection portion 122 is an area where a circular half-celestial sphere image that is fisheye-projected onto the solid-state imaging device 42 by the imaging lens 16 is displayed. The imaging lens 16 is adjusted so that the center of the pixel region 121 and the center of the effective projection portion 122 are aligned.

The outermost periphery of the circular effective projection portion 122 indicates a position with an FOV (Field of view) angle of 180°. When the user is looking at the center of the horizontal and vertical directions, the angle of the target field of view 125, which is the area to be imaged and recorded, from the center of the effective projection portion 122 is half the angle of 90°.

The imaging lens 16 of this embodiment can also introduce light rays outside the effective projection portion 122, and can project light rays up to a maximum FOV angle of 192° onto the solid-state image pickup device 42 as a fisheye. However, when the effective projection area 122 is exceeded, the optical performance is greatly deteriorated, such as the resolving power being extremely reduced, the amount of light being reduced, and the distortion becoming stronger. Therefore, in this embodiment, the recording area is an example in which the image in the observation direction is cut out only from the image projected on the pixel area 121 (hereinafter simply referred to as the ultra-wide-angle image) of the half-celestial image displayed on the effective projection unit 122. will explain.

In this embodiment, since the size of the effective projection portion 122 in the vertical direction is larger than the size of the short side of the pixel region 121, the images at the upper and lower ends of the effective projection portion 122 are out of the pixel region 121, but the present invention is not limited to this. . For example, the configuration of the imaging lens 16 may be changed so that the entire effective projection portion 122 may be designed to fit within the pixel region 121 .

An invalid pixel area 123 is a pixel area that is not included in the effective projection area 122 in the pixel area 121 .

The target field of view 125 is an area indicating the range of cutting out the image in the viewing direction of the user from the ultra-wide-angle image, and is defined by a preset left, right, up, and down angle of view centered on the viewing direction (here, 45°, FOV angle of 90°). ). In the example of FIG. 11A, since the user faces the front, the center of the target field of view 125 is the observation direction vo, which is the center of the effective projection portion 122 .

The ultra-wide-angle image shown in FIG. 11A includes a subject A131 that is a child, a subject B132 that is a staircase that the child who is subject A is trying to climb, and a subject C133 that is a locomotive-shaped playground equipment.

Next, FIG. 7D shows the recording direction/range determination processing in step S300 that is executed to obtain the image in the viewing direction from the super-wide-angle image described using FIG. 11A. This process will be described below with reference to FIGS.

In step S<b>301 , the preset field angle setting value ang is obtained by reading from the primary memory 103 .

In the present embodiment, all the view angles, 45°, 90°, 110°, and 130°, which enable the image clipping/development processing unit 50 to clip the video in the observation direction from the super-wide-angle image, are the view angle set values ang is stored in the built-in nonvolatile memory 102 as . Also, in one of steps S103, S106, and S108, one of the field angle setting values ang stored in the built-in nonvolatile memory 102 is set and stored in the primary memory 103. FIG.

In step S301, the observation direction vi determined in step S212 is determined as the recording direction, and the image of the target visual field 125 cut out from the ultra-wide-angle image with the above-obtained view angle setting value ang centered on this is processed as a primary image. Save in memory 103 .

For example, when the field angle setting value ang is 90° and the observation direction vo (vector information [0°, 0°]) is detected in the face direction detection process (FIG. 7C), the center O of the effective projection unit 122 A target visual field 125 (FIG. 11A) is set to a range of 45° left and right and 45° up and down from the center. That is, the overall control CPU 101 sets the angle of the face direction detected by the face direction detection unit 20 as the viewing direction vi, which is vector information indicating the relative position with respect to the ultra-wide-angle video.

Here, in the case of the observation direction vo, since the influence of the optical distortion by the imaging lens 16 can be almost ignored, the shape of the set target field of view 125 can be used as it is for the target field of view 125o (FIG. 12A) after distortion conversion in step S303, which will be described later. shape. Hereinafter, the target field of view 125 after distortion conversion in the case of the observation direction vi is referred to as a target field of view 125i.

Next, in step S302, a preset anti-vibration level is read out from the primary memory 103 to obtain it.

In this embodiment, as described above, the image stabilization level is set in one of steps S103, S106, and S108 and stored in the primary memory 103. FIG.

Further, in step S302, the anti-vibration spare pixel quantity Pis is set based on the obtained anti-vibration level.

In the anti-vibration processing, an image that follows the image in the direction opposite to the direction of the blur is obtained by following the blur amount of the photographing/detecting unit 10 . Therefore, in this embodiment, a spare area necessary for image stabilization is provided around the target visual field 125i.

Further, in this embodiment, the built-in non-volatile memory 102 stores a table that holds the value of the number of anti-shake pixels Pis associated with each anti-shake level. For example, if the image stabilization level is "medium", a spare pixel area of 100 pixels, which is the number of image stabilization spare pixels Pis read from the table, is set as the spare area.

FIG. 12E is a diagram showing an example in which a preliminary area is provided around the target visual field 125o shown in FIG. 12A. Here, the case where the image stabilization level is "medium", that is, the image stabilization preliminary pixel amount Pis is 100 pixels will be described.

As shown in FIG. 12E, the dashed line portion with a margin (preliminary area) of 100 pixels corresponding to the amount of anti-shake spare pixels Pis on each side of the target visual field 125o is a spare pixel frame 126o for anti-shake.

For simplicity of explanation, FIGS. 12A and 12E have explained the case where the observation direction vi coincides with the center O of the effective projection section 122 (the optical axis center of the imaging lens 16). However, as will be explained in the following steps, if the viewing direction vi is in the periphery of the effective projection portion 122, it will be affected by optical distortion and must be transformed.

In step S303, the shape of the target field of view 125 set in step S301 is corrected (distorted) in consideration of the observation direction vi and the optical characteristics of the imaging lens 16 to generate a target field of view 125i. Similarly, the number of spare pixels for vibration reduction Pis set in step S302 is also corrected in consideration of the viewing direction vi and the optical characteristics of the imaging lens 16. FIG.

For example, assume that the angle of view setting value ang is 90° and the user is observing 45° to the right of the center o. In this case, the observation direction vi determined in step S212 is the observation direction vr (vector information [45°, 0°]), and the target field of view 125 is the range of 45° left and right and 45° up and down centered on the observation direction vr. becomes. However, considering the optical characteristics of the imaging lens 16, the target field of view 125 is corrected to the target field of view 125r shown in FIG. 12B.

As shown in FIG. 12B, the target field of view 125r widens toward the periphery of the effective projection portion 122, and the position of the viewing direction vr is slightly inside the center of the target field of view 125r. This is because, in this embodiment, the imaging lens 16 has an optical design similar to a stereographic fisheye. Incidentally, if the imaging lens 16 is designed with an equidistant projection fisheye, an equisolid angle projection fisheye, or an orthogonal projection fisheye, etc., the relationship will change. performed for

FIG. 12F is a diagram showing an example in which a spare area having the same image stabilization level of "middle" as that of the spare area shown in FIG. 12E is provided around the target field of view 125r shown in FIG. 12B.

In the anti-vibration spare pixel frame 126o (FIG. 12E), a margin of 100 pixels, which is the number of anti-vibration spare pixels Pis, is set in each of the top, bottom, left, and right of the target visual field 125o. On the other hand, in the anti-vibration spare pixel frame 126r (FIG. 12F), the number of anti-vibration spare pixels Pis is corrected and increased toward the periphery of the effective projection portion 122. FIG.

In this way, the shape of the preliminary area necessary for vibration reduction provided around the target field of view 125r is also the same as the shape of the target visual field 125r, as shown in the vibration reduction preliminary pixel frame 126r in FIG. 12F. The amount of correction increases as it goes to the part. This is also because, in this embodiment, the imaging lens 16 has an optical design similar to a stereoscopic projection fisheye. If the image pickup lens 16 is designed with an equidistant projection fisheye, an equisolid angle projection fisheye, orthographic projection fisheye, the relationship will change, so correction according to the optical characteristics is necessary for vibration reduction. This is done for the spare pixel frame 126r.

The process of sequentially switching the shape of the target field of view 125 and its preliminary area in consideration of the optical characteristics of the imaging lens 16 executed in step S303 is a complicated process. Therefore, in this embodiment, the process of step S303 is executed using a table in the built-in non-volatile memory 102 that holds the shape of the target field of view 125i for each observation direction vi and its preliminary area. Incidentally, depending on the optical design of the imaging lens 16 mentioned above, an arithmetic expression may be stored in the overall control CPU 101, and the optical distortion value may be calculated by the arithmetic expression.

In step S304, the position and size of the video recording frame are calculated.

As described above, in step S303, a preliminary area necessary for vibration reduction is provided around the target visual field 125i, and this is calculated as a vibration reduction preliminary pixel frame 126i. However, depending on the position in the viewing direction vi, its shape becomes quite special, such as the anti-vibration spare pixel frame 126r.

The overall control CPU 101 can cut out an image by performing development processing only for such a special shape range. However, it is not common to use a non-rectangular image when recording as image data in step S600 or transferring to display device 800 in step S700. Therefore, in step S304, the position and size of a rectangular video recording frame 127i that includes the entire anti-shake spare pixel frame 126i are calculated.

FIG. 12F shows a video recording frame 127r indicated by a dashed line, which is calculated in step S304 for the anti-shake spare pixel frame 126r.

In step S305, the position and size of the video recording frame 127i calculated in step S304 are recorded in the primary memory 103. FIG.

In this embodiment, the upper left coordinates Xi, Yi of the image recording frame 127i in the ultra-wide-angle image are recorded as the position of the image recording frame 127i, and the horizontal width WXi and the vertical width WXi of the image recording frame 127i from the coordinates Xi, Yi are recorded. The width WYi is recorded as the size of the video recording frame 127i. For example, for the video recording frame 127r shown in FIG. 12F, the illustrated coordinates Xr, Yr, horizontal width WXr, and vertical width WYr are recorded in step S305. The coordinates Xi and Yi are XY coordinates with a predetermined reference point, specifically, the optical center of the imaging lens 16 as the origin.

When the anti-shake spare pixel frame 126i and the video recording frame 127i are determined in this manner, the subroutine of step S300 shown in FIG. 7D is exited.

In the description so far, for the sake of simplification of the explanation of the complicated optical distortion conversion, the observation direction vi is taken as an example of the observation direction including the horizontal 0°, that is, the observation direction vo (vector information [0°, 0°] ) and observation direction vr (vector information [45°, 0°]). However, in practice, the viewing direction vi of the user varies. Therefore, the recording area development processing executed in such a case will be described below.

For example, the target visual field 125l when the field angle setting value ang is 90° and the observation direction vl[−42°, −40°] is as shown in FIG. 12C.

Also, even if the observation direction vl (vector information [−42°, −40°]) is the same as that of the target field of view 125l, when the angle of view setting value ang is 45°, as shown in FIG. 12D, the target field of view 125l The target visual field 128l is one size smaller. Furthermore, for the target field of view 128l, a vibration reduction spare pixel frame 129l and a video recording frame 130l are set as shown in FIG. 12G.

Step S400 is the basic operation of imaging, and since a general sequence of the imaging unit 40 is used, the details will be left to other literatures and will not be described here. In this embodiment, the imaging signal processing circuit 43 in the imaging unit 40 converts the signal output from the solid-state imaging device 42 in a specific output format (examples of standards: MIPI, SLVS) to a general sensor. It also performs a process of correcting the imaging data to read-out method.

When the mode selected by the imaging mode switch 12 is the moving image mode, the imaging unit 40 starts recording when the start switch 14 is pressed. After that, when the stop switch 15 is pressed, the recording ends. On the other hand, when the mode selected by the imaging mode switch 12 is the still image mode, the imaging unit 40 takes a still image each time the start switch 14 is pressed.

FIG. 7E is a flowchart of a subroutine of recording area development processing in step S500 of FIG. 7A.

In step S501, raw data of the entire area of the imaging data (ultra-wide-angle video) generated by the imaging unit 40 in step S400 is obtained and input to an image capture unit called a head unit (not shown) of the overall control CPU 101.

Next, in step S502, based on the coordinates Xi, Yi, width WXi, and height WYi recorded in the primary memory 103 in step S305, a portion of the video recording frame 127i is cut out from the ultra-wide-angle video acquired in step S501. After this cropping, the crop development process (FIG. 7F) consisting of steps S503 to S508 executed below is started only for the pixels within the anti-vibration spare pixel frame 126i. As a result, the amount of calculation can be greatly reduced compared to the case where development processing is performed on the entire area of the super-wide-angle image read in step S501, and the calculation time and power can be reduced.

As shown in FIG. 7F, when the mode selected by the imaging mode switch 12 is the moving image mode, the processes of steps S200 and S300 and the process of step S400 are executed in parallel at the same or different frame rates. be done. That is, each time the raw data of the entire area for one frame generated by the imaging unit 40 is acquired, cropping is performed based on the coordinates Xi and Yi, the horizontal width WXi, and the vertical width WYi recorded in the primary memory 103 at that time. Development processing is performed.

When the crop development process for the pixels in the anti-vibration spare pixel frame 126i is started, first, in step S503, color interpolation is performed to complement the color pixel information arranged in the Bayer array.

Thereafter, after adjusting the white balance in step S504, color conversion is performed in step S505.

In step S506, gamma correction is performed to correct the gradation according to a preset gamma correction value.

In step S507, edge enhancement is performed in accordance with the image size.

In step S508, the data is converted into a data format that can be temporarily stored by performing compression and other processing, and after recording in the primary memory 103, this subroutine is exited. The details of the data format that can be temporarily stored will be described later.

It should be noted that the order of the cropping development processing executed in steps S503 to S508 and the presence or absence of processing may be determined in accordance with the camera system, and are not intended to limit the present invention.

Also, when the moving image mode is selected, the processing from steps S200 to S500 is repeatedly executed until the recording ends.

According to this process, the amount of calculation can be greatly reduced compared to the case where the development process is performed on the entire area read in step S501. Therefore, an inexpensive and low power consumption microcomputer can be used as the overall control CPU 101, heat generation in the overall control CPU 101 can be suppressed, and the battery 94 can last longer.

Further, in this embodiment, in order to lighten the control load of the overall control CPU 101, the image optical correction processing (step S800 in FIG. 7A) and image stabilization processing (step S900 in FIG. 7A) are not performed in the camera body 1, and the display After transferring to the device 800 , the display device control unit 801 performs the processing. Therefore, if only data of an image partially clipped from a projected ultra-wide-angle image is sent to the display device 800, optical correction processing and vibration reduction processing cannot be performed. In other words, the data of the clipped image alone does not have the position information used for substituting it into the formula for the optical correction processing or referring to the correction table for the image stabilizing processing. 800 cannot run correctly. Therefore, in this embodiment, not only data of the clipped image but also correction data including information on the clipping position of the image from the ultra-wide-angle image are transmitted from the camera body 1 to the display device 800 .

If the clipped image is a still image, even if the data of the still image and the correction data are separately transmitted to the display device 800, the data of the still image and the correction data are in one-to-one correspondence. In 800, correct optical correction processing and image stabilizing processing can be performed. On the other hand, if the clipped image is a moving image, if the data of the moving image and the correction data are separately transmitted to the display device 800, it is determined which correction data for each frame of the moving image is the transmitted correction data. becomes difficult. In particular, if the clock rate of the overall control CPU 101 in the camera body 1 and the clock rate of the display device control unit 801 in the display device 800 are slightly different, the difference between the overall control CPU 101 and the display device control unit 801 will occur after several minutes of video shooting. out of sync. As a result, the display device control unit 801 corrects the frame to be processed with correction data different from the correction data corresponding thereto.

Therefore, in the present embodiment, when data of a moving image cut out from the camera body 1 is transmitted to the display device 800, the correction data is appropriately given to the data of the moving image. The method will be described below.

FIG. 14 is a flowchart of a subroutine of primary recording processing in step S600 of FIG. 7A. This processing will be described below with reference to FIG. FIG. 14 shows processing when the mode selected by the imaging mode switch 12 is the moving image mode. If the selected mode is still image mode, this process starts from step S601 and ends when step S606 ends.

In step S601a, the overall control CPU 101 reads out one frame image for which steps S601 to S606 have not been processed from the moving image developed in the recording area development process (FIG. 7E). Also, the overall control CPU 101 generates correction data, which is metadata of the read frame.

In step S601, the overall control CPU 101 attaches information about the clipping position of the image of the frame read out in step S600 to the correction data. The information attached here is the coordinates Xi and Yi of the video recording frame 127i acquired in step S305. The information attached here may be vector information indicating the viewing direction Vi.

In step S602, the overall control CPU 101 acquires an optical correction value. The optical correction value is the optical distortion value set in step S303. Alternatively, correction values corresponding to lens optical characteristics such as peripheral light amount correction values and diffraction correction may be used.

In step S603, the overall control CPU 101 attaches the optical correction value used for distortion conversion in step S602 to the correction data.

In step S604, the overall control CPU 101 determines whether or not the mode is anti-vibration mode.
Specifically, if the preset anti-shake mode is "medium" or "strong," it is determined that the anti-shake mode is on, and the process proceeds to step S605. On the other hand, if the anti-vibration mode set in advance is "off", it is determined that the anti-vibration mode is not set, and the process proceeds to step S606. The reason why step S605 is skipped when the anti-shake mode is "OFF" is that the amount of data calculated by the overall control CPU 101 and the amount of data during wireless transmission can be reduced by skipping. This is because power consumption and heat generation can also be reduced. Although the reduction of the data used for image stabilizing processing has been described here, it is also possible to reduce the peripheral illumination correction value and the data regarding the presence/absence of analysis correction included in the optical correction value acquired in step S602.

In this embodiment, the anti-vibration mode is set in advance by the user's operation on the display device 800 , but may be set as an initial setting of the camera body 1 . Also, in the case of a camera system that switches between the presence and absence of vibration reduction processing after transfer to the display device 800, step S604 is omitted, and the process proceeds directly from step S603 to step S605.

In step S605, the overall control CPU 101 attaches the anti-vibration mode acquired in step S302 and the gyro data during moving image shooting associated with the frame read out in step S600 in the primary memory 813 to the correction data.

In step S606, the video file 1000 (FIG. 15) is updated with data obtained by encoding the frame image data read in step S600 and the correction data to which various data are attached in steps S601 to S605. Note that when the first frame of the moving image is read in step S601a, the image file 1000 is generated in step S606.

In step S607, it is determined whether reading of images of all frames of the moving image developed by the recording area development process (FIG. 7E) has been completed. If not, the process returns to step S601a. On the other hand, if it has ended, this subroutine is exited. The generated video file 1000 is saved in the built-in non-volatile memory 102 . It may be stored not only in the primary memory 813 and the built-in non-volatile memory 102 described above, but also in the large-capacity non-volatile memory 51 . Alternatively, the generated video file 1000 may be immediately transferred to the display device 800 (step S700 in FIG. 7A), transferred to the display device 800, and then stored in the primary memory 813 thereof.

Here, in this embodiment, "encoding" refers to combining video data and correction data into a single file. Can be compressed.

FIG. 15 is a diagram showing the data structure of the video file 1000. As shown in FIG.

A video file 1000 is composed of a header 1001 and frames 1002 . A frame 1002 is composed of a frame data set in which an image of each frame constituting a moving image and a frame meta corresponding thereto are set. That is, the frame 1002 has frame data sets for the total number of frames of the moving image.

In this embodiment, the frame meta is information encoded with correction data to which a clipping position (intra-video position information), an optical correction value, and gyro data are attached as necessary, but is not limited to this. For example, depending on the imaging mode selected by the imaging mode switch 12, the information amount of the frame meta can be changed by attaching other information to the frame meta or by deleting information in the frame meta. Also good.

In the header 1001, the offset value or start address of each frame to the frame data set is recorded. Alternatively, metadata such as time and size corresponding to the video file 1000 may be saved.

In this way, in the primary recording process (FIG. 14), the image file 1000 in which each frame of the moving image developed in the recording area development process (FIG. 7E) and its metadata are set is transferred to the display device 800. be done. Therefore, even if the clock rate of the overall control CPU 101 of the camera body 1 and the clock rate of the display device control section 801 of the display device 800 are slightly different, the display device control section 801 can correct the moving image developed by the camera body 1. process can be performed reliably.

In this embodiment, the optical correction value is included in the frame meta, but the optical correction value may be applied to the entire image.

FIG. 16 is a flowchart of a subroutine of transfer processing to display device 800 in step S700 of FIG. 7A. FIG. 16 shows processing when the mode selected by the imaging mode switch 12 is the moving image mode. Note that if the selected mode is the still image mode, this process starts from step S702.

In step S701, it is determined whether or not recording of the moving image by the imaging unit 40 (step S400) is completed, or whether recording is in progress. Here, when a moving image is being recorded (moving image is being captured), recording area development processing (step S500) for each frame and updating of the image file 1000 (step S606) in the primary recording processing (step S600) are sequentially performed. It will be in the state of being done. Since wireless transfer has a large power load, if it is performed in parallel with recording, a large battery capacity of the battery 94 is required, or heat generation countermeasures need to be taken separately. Also, from the point of view of computing power, performing wireless transfer in parallel with video recording increases the computing load, so it is necessary to prepare a high-spec overall control CPU 101, which increases the cost. In view of this, in this embodiment, after waiting for the end of recording of the moving image (YES in step S701), the process proceeds to step S702 to establish connection with the display device 800. FIG. However, if the camera system of the present embodiment has sufficient power supplied from the battery 94 and does not require a separate countermeasure against heat generation, the display device 800 may be set in advance when the camera body 1 is activated or before recording is started. You can connect with

In step S 702 , a connection is established with the display device 800 via the high-speed wireless unit 72 in order to transfer the video file 1000 with a large amount of data to the display device 800 . The low-power wireless unit 71 is used for transferring low-resolution video (or video) for checking the angle of view to the display device 800 and for transmitting and receiving various setting values to and from the display device 800. Since it takes time to transfer the video file 1000, it is not used.

In step S703, the video file 1000 is transferred to the display device 800 via the high-speed wireless unit 72, and when the transfer is completed, the process advances to step S704, closes the connection with the display device 800, and exits from this subroutine.

Up to this point, we have explained the case of transferring a single video file containing images of all frames of a single moving image. good. If the video file has the data structure shown in FIG. 15, even if one moving image is transferred to the display device 800 as a plurality of video files, the moving image can be corrected on the display device 800 without timing deviation from the correction data. It becomes possible.

FIG. 17 is a flowchart of a subroutine of optical correction processing in step S800 of FIG. 7A. This processing will be described below with reference to FIG. Note that, as described above, this processing is processing executed by the display device control unit 801 of the display device 800 .

In step S801, first, the display device control unit 801 receives the video file 1000 from the camera body 1 transferred in the transfer processing to the display device 800 (step S700). After that, the display device control unit 801 acquires the optical correction values extracted from the received video file 1000 .

Subsequently, in step S<b>802 , the display device control unit 801 acquires a video (one frame image obtained by capturing a moving image) from the video file 1000 .

In step S<b>803 , the display device control unit 801 optically corrects the image acquired in step S<b>802 using the optical correction value acquired in step S<b>801 , and stores the corrected image in the primary memory 813 . When cutting out the image acquired in step S802 when optical correction is performed, the image is cut out in an image range narrower than the development range (target visual field 125i) determined in step S303 (cutout development region).

FIG. 18 is a diagram for explaining a case where distortion aberration correction is performed in step S803 of FIG.

FIG. 18(a) is a diagram showing the position of the subject 1401 as seen by the user's naked eye during imaging, and FIG.

FIG. 18(c) shows the developed area 1402 in the image of FIG. 18(b). Here, the development area 1402 is the clipped development area described above.

FIG. 18(d) is a diagram showing a clipped development area in which the image of the development area 1402 is clipped, and FIG. 18(e) is a diagram showing an image obtained by correcting the distortion of the clipped development area in FIG. 18(d). is. Since the clipping process is performed when correcting the distortion of the clipped developed image, the angle of view of the image shown in FIG.

FIG. 19 is a flow chart of a subroutine of image stabilizing processing in step S900 of FIG. 7A. This processing will be described below with reference to FIG. Note that, as described above, this processing is processing executed by the display device control unit 801 of the display device 800 .

In step S901, from the frame meta of the video file 1000, the gyro data of the current frame and the previous frame, and the shake amount V _n−1 ^Det calculated for the previous frame in step S902, which will be described later, are acquired. After that, an approximate shake amount V _n ^Pre is calculated from these pieces of information. In this embodiment, the current frame is the frame currently being processed, and the previous frame is the frame one frame before the current frame.

In step S902, a detailed blur amount _VnDet is ^obtained from the image. The blur amount is detected by calculating how much the feature point of the image of the current frame has moved from the previous frame.

A known method can be adopted for extracting feature points. For example, a luminance information image is generated by extracting only the luminance information of the image of the frame, and the image obtained by shifting the image by one to several pixels is subtracted from the original image, and the pixels whose absolute value is greater than the threshold value are extracted as feature points. You may Further, an image obtained by applying a high-pass filter to the luminance information image may be subtracted from the original luminance information image, and the extracted edge may be extracted as a feature point.

Differences are calculated a plurality of times while the luminance information images of the current frame and the previous frame are shifted by one to several pixels, and the movement amount is calculated by calculating the position where the difference in the pixel of the feature point becomes small.

Since a plurality of feature points are required as described later, it is preferable to extract feature points by dividing each image of the current frame and the previous frame into a plurality of blocks. Generally, 4×3=12 blocks or 96×64 blocks are preferable, although the block division depends on the number of pixels and the aspect ratio of the image. If the number of blocks is too small, it will be impossible to accurately correct trapezoids and rotations in the direction of the optical axis due to tilting of the photographing unit 40 of the camera body 1. However, if the number of blocks is too large, the size of one block will be small and feature points will be close. This is because it contains an error because For this reason, the optimal number of blocks is appropriately selected depending on the number of pixels, the ease of finding characteristic points, the angle of view of the subject, and the like.

To calculate the amount of movement, it is necessary to shift the luminance information images of the current frame and the previous frame by one to several pixels and calculate the difference a plurality of times, resulting in a large amount of calculation. Therefore, in order to calculate how many pixels the actual amount of movement is shifted from the blur amount V _n ^Pre , it is possible to greatly reduce the amount of calculation by calculating the difference only in the vicinity thereof.

Next, in step S903, image stabilization correction is performed using the detailed blur amount _VnDet acquired in step S902, and then this subroutine is ^exited .

Conventionally known image stabilization methods include Euclidean transformation capable of rotation and translation, affine transformation capable of such transformation, and projective transformation capable of trapezoidal correction.

Movement and rotation along the X-axis and Y-axis can be performed by Euclidean transformation, but when the photographing unit 40 of the camera body 1 actually captures an image, there are camera shakes in the front-rear direction and in the pan/tilt direction. . Therefore, in the present embodiment, image stabilizing correction is performed using affine transformation capable of correcting enlargement, skew, and the like. In the affine transformation, when the coordinates (x, y) of the reference feature point move to the coordinates (x', y'), it is represented by the following equation 100.

The affine coefficient of the 3×3 matrix of Equation 100 can be calculated if at least three feature point shifts can be detected. However, when the detected feature points are close to each other or on a straight line, image stabilization correction is inaccurate at locations farther than the feature points or away from the straight line. Therefore, it is preferable to select feature points that are far apart from each other and do not lie on a straight line. Therefore, when a plurality of feature points are detected, feature points close to each other are omitted and the rest are normalized by the least squares method.

FIG. 18(f) is a diagram showing an image obtained by subjecting the distortion-corrected image shown in FIG. 18(e) to the anti-vibration correction in step S903. Since clipping processing is performed during image stabilizing correction, the angle of view of the image shown in FIG. 18(f) is smaller than that of the image shown in FIG. 18(e).

By performing such anti-vibration processing, it is possible to obtain a high-quality image in which blur is corrected.

A series of operations performed by the camera body 1 and the display device 800 included in the camera system of this embodiment have been described above.

When the user turns on the power switch 11, selects the moving image mode with the imaging mode switch 12, and observes the front without looking up, down, left, or right, first, the face direction detection unit 20 detects the observation direction. Detect vo(vector information [0°, 0°]) (FIG. 12A). After that, the recording direction/angle of view determination unit 30 cuts out the image (FIG. 11B) of the target field of view 125o shown in FIG.

After that, when the user starts observing, for example, the child (subject A131) in FIG. °]) (FIG. 11C). After that, the recording direction/angle of view determination unit 30 cuts out the image of the target field of view 125l (FIG. 11C) from the ultra-wide-angle image captured by the image capturing unit 40 .

In steps S800 and S900, the display device 800 performs optical correction processing and vibration reduction processing on images cut out in various shapes according to the viewing direction. As a result, even if the overall control CPU 101 of the camera body 1 is of low spec, even if the image of the target field of view 125l (FIG. 11C), for example, which has a large distortion, is cut out, the child (subject A 131) is centered as shown in FIG. 11D. An image corrected for distortion and shaking can be obtained. In other words, the user can obtain an image in which his observation direction is captured without touching the camera body 1 except for turning on the power switch 11 and selecting the mode with the imaging mode switch 12 .

The preset mode will now be described. As described above, since the camera body 1 is a small wearable device, the camera body 1 is not provided with operation switches, setting screens, and the like for changing detailed settings. Therefore, the detailed settings of the camera body 1 are changed using an external device such as the display device 800 (the setting screen of the display device 800 (FIG. 13) in this embodiment).

For example, consider a case where it is desired to continuously capture the same moving image at a field angle of 90° and a field angle of 45°. Since the angle of view is set to 90° in the normal moving image mode, when performing such image capturing, first, after image capturing in the normal moving image mode, the moving image capturing is finished once, and the display device 800 is moved to the camera body 1. It is necessary to switch to the setting screen and switch the angle of view to 45 degrees. However, during continuous imaging, such operations on the display device 800 are troublesome, and the user may miss the desired image.

On the other hand, if the preset mode is set in advance to a mode for capturing a moving image with an angle of view of 45°, after capturing a moving image with an angle of view of 90°, all you have to do is slide the imaging mode switch 12 to "Pri". , can be immediately changed to a zoomed-in moving image with an angle of view of 45°. That is, the user does not need to interrupt the current imaging action and perform the above-described troublesome operations.

The contents to be set in the preset mode include not only changing the angle of view, but also the image stabilization level specified by "strong", "medium", "off", etc., and voice recognition settings not described in this embodiment. may be included.

For example, when the user switches from the moving image mode to the preset mode with the imaging mode switch 12 while continuing to observe the child (subject A131) in the previous imaging situation, the field angle setting value ang is changed from 90° to 45°. be. In this case, the recording direction/angle of view determination unit 30 cuts out the image of the target field of view 128l indicated by the dotted line frame shown in FIG.

Even in the preset mode, optical correction processing and image stabilization processing are performed in display device 800 in steps S800 and S900. As a result, even if the general control CPU 101 of the camera body 1 has a low spec, it is possible to obtain a zoomed image centering on the child (subject A131) as shown in FIG. Although the example in which the angle of view setting value ang is changed from 90° to 45° in the moving image mode has been described, the same applies to the still image mode. The same applies when the set angle of view value ang for moving images is 90 degrees and the set angle of view ang for still images is 45 degrees.

In this manner, the user can obtain a zoomed image in which the observation direction of the user is captured only by switching modes with the imaging mode switch 12 on the camera body 1 .

In this embodiment, the case where the face direction detection section 20 and the photographing section 40 are integrally configured in the camera body 1 has been described. It is not limited to this as long as it is worn and the imaging unit 40 is worn on the user's body. For example, the imaging/detecting unit 10 of this embodiment can be installed on the shoulder or on the abdomen. However, if the imaging unit 40 is placed on the right shoulder, the subject on the left side may be blocked by the head. preferable.

(Example 2)
Transfer time is a problem when transferring an image captured by an imaging device to a display device such as a smartphone for display.

For example, when an imaging device and a display device are connected by WiFi (IEEE802.11a or IEEE802.11g) and images shot at 30 Mbps are transferred, a 10-minute moving image takes about 5 minutes and 40 seconds.

Since the actual transfer speed is often lower than the theoretical value, it takes longer than that.

Therefore, in the second embodiment, a transfer range is set by using a low-quality image whose data size (number of pixels, bit rate, or frame rate) is smaller than that of a captured high-quality image. To shorten the transfer time of video by transferring only high-definition video.

This embodiment will be described as being basically derived from the first embodiment.

Therefore, in the configuration of the camera system (video transfer system) of the second embodiment, the same components as those of the camera system of the first embodiment are denoted by the same reference numerals, and overlapping descriptions are omitted. Details will be added and explained.

First, the camera body 1 in Example 2 will be described.

FIG. 23 is a functional block diagram of the camera body 1 according to the second embodiment.

The camera body 1 (imaging device) of the second embodiment includes an image recording section 2301, an image quality conversion unit, and an image recording section 2301 instead of the primary recording section 60, the transmission section 70, and the other control section 111 of the camera body 1 of the first embodiment shown in FIG. It has a section 2302 and a communication section 2304 .

The image recording unit 2301 records a high-quality captured image (high-quality image, first image) developed and cut out by the image cut-out/development processing unit 50 and a low-quality image (second image) by the image quality conversion unit 2302 . Record the captured video that has been converted to The image recording unit 2301 is composed of the primary memory 103 or the large-capacity nonvolatile memory 51 .

The image recording unit 2301 (recording means) also transfers part or all of the recorded video to the communication unit 2304 at the necessary timing.

The image quality conversion unit 2302 (conversion means) converts the shot image that has undergone the development/cutout processing by the image cutout/development processing unit 50 into a low quality image. The conversion to low-quality video in the image quality conversion unit 2302 is realized by executing at least one of lowering the resolution of the captured video, lowering the bit rate, and lowering the frame rate.

The processing of the image quality conversion unit 2302 may be executed by the overall control CPU 101, or may be executed by providing an image processing circuit separately.

The communication unit 2304 (first communication means) is the same as the transmission unit 70, but in addition to transmitting video, it can also receive video clipping sections (editing information) from the display device 800. .

Next, a transfer setting screen displayed on the display unit 803 (display means) of the display device 800 (display device) according to the second embodiment will be described with reference to FIG.

The user uses the transfer setting screen to set the transfer interval of the high-quality video transferred from the camera body 1 to the display device 800 .

FIG. 25A is a schematic diagram showing an example of the initial state of the transfer setting screen in the second embodiment.

As shown in FIG. 25A, the transfer setting screen on the display unit 803 of the display device 800 displays a message 2520, frame images 2501 to 2512, and a transfer start button 2521 in the initial state.

As shown in FIG. 25A, the user cannot press the transfer start button 2521 while the transfer start button 2521 is grayed out.

The frame images 2501 to 2512 are obtained by dividing the low-quality video transmitted from the camera body 1 by the display device control unit 801 (extracting means/display means) in predetermined periods, and extracting the frame images from each of the divided videos. They are representative images and are displayed in chronological order. In this embodiment, this predetermined period is set to 10 seconds, but it is not limited to this.

For example, the frame image 2501 is an image representing 10 seconds from the start of the low image quality image received from the camera body 1, and the frame image 2502 is an image representing 10 to 20 seconds. A frame image 2503 and subsequent frames are images representing ten seconds in the same manner, and a frame image 2512 is an image representing the last ten seconds of the low image quality video received from the camera body 1 .

That is, in FIG. 25A, 120 seconds of low-quality video are represented by 12 representative images. As a result, the user can confirm the content of the 120-second low-quality video without reproducing it on the display device 800 .

In this embodiment, the image of the first frame of each period is used as the representative image, but the image of the frame that best represents each period may be used as the representative image by analyzing the video.

Also shown in FIG. 25(a) is a message 2520 prompting the user to select a transfer start position.

The user can set the transfer start position by selecting one of the frame images 2501 to 2512 .

FIG. 25(b) is a schematic diagram showing the state of the transfer setting screen when the user selects the frame image 2504 as the transfer start position on the transfer setting screen shown in FIG. 25(a).

As shown in FIG. 25B, a check mark is superimposed on the frame image 2504 . Thereby, the user can confirm that the frame image 2504 has been set on the display device 800 as the transfer start position by the display device control section 801 (edit information setting means).

Also, the message 2520 is changed to a message prompting the user to select the transfer end position.

FIG. 25(c) is a schematic diagram showing the state of the transfer setting screen when the user selects the frame image 2506 as the transfer end position on the transfer setting screen shown in FIG. 25(b).

As shown in FIG. 25C, not only the frame image 2504 but also the frame image 2506 are superimposed with check marks. Thereby, the user can confirm that the frame image 2506 has been set on the display device 800 as the transfer end position by the display device control section 801 (editing information setting means).

In addition, the transfer start button 2521 grayed out in FIGS. 25(a) and (b) is cleared in gray out in FIG. 25(c). As a result, the user can understand that the transfer start button 2521 can be pressed.

Next, video transfer processing of the camera system according to the second embodiment will be described.

FIG. 24 is a flowchart of image transfer processing of the camera system according to the second embodiment.

For ease of explanation, in this flowchart, the processing in the display device 800 is shown on the left side, and the processing by the camera body 1 is shown on the right side.

That is, each process of steps S2401 to S2407 is realized in display device 800 by display device control unit 801 executing a program stored in built-in nonvolatile memory 812. FIG. On the other hand, each process of steps S2409 to S2412 is realized in the camera body 1 by the overall control CPU 101 executing a program stored in the built-in non-volatile memory 102. FIG.

Also, this flowchart starts from a state in which the display device 800 and the camera body 1 are already connected by high-speed wireless.

When the display device control unit 801 receives a video transfer request from the camera body 1 , first, the display device control unit 801 transmits a low-quality video acquisition request to the camera body 1 . Accordingly, the display device control section 801 uses the high-speed wireless unit 872 or the low-power wireless unit 871 (second communication means) to obtain a low-quality image from the camera body 1 (step S2401). When the acquisition of the low image quality video is completed, the display device control unit 801 proceeds to step S2402.

Overall control CPU 101 uses communication unit 2304 to transmit the low-quality video to display device 800 in response to the acquisition request transmitted from display device 800 in step S2401 (step S2409). The video transmitted in step S2409 is the low image quality video recorded in the image recording unit 2301. FIG.

In step S2402, the display device control unit 801 displays a transfer setting screen, and then advances to step S2403 to wait for the user to press the transfer start button 2521 to instruct the start of transfer.

When the user presses the transfer start button 2521 to instruct the start of image transfer (YES in step S2403), the display device control unit 801 advances to step S2404 to calculate the transfer start time of the high-quality video.

For example, when the transfer start button 2521 is pressed on the transfer setting screen shown in FIG. The position of 30 seconds is calculated as the transfer start time.

Next, the display device control unit 801 calculates the transfer end time of the high-quality video (step S2405).

For example, when the transfer start button 2521 is pressed on the transfer setting screen shown in FIG. The position of 60 seconds is calculated as the transfer end time.

Next, the display device control section 801 transmits the edit information using the high-speed wireless unit 872 or the low-power wireless unit 871 (second communication means) (step S2406). Specifically, as edit information, the transfer start time and transfer end time (time cut-out section) calculated in steps S2404 and S2405 are transmitted to the camera body 1 .

Overall control CPU 101 uses communication unit 2304 to receive the transfer start time and transfer end time transmitted from display device 800 in step S2406 (step S2410).

Overall control CPU 101 (image editing means) then cuts out a predetermined section from the high-quality video based on the transfer start time and transfer end time received in step S2410 (step S2411). In other words, only the image of the period specified by the transfer start time and the transfer end time is clipped to generate a high-quality image with a small size.

Overall control CPU 101 then transmits the high-quality video cut out in step S2411 to display device 800 using high-speed radio unit 872 and communication section 2304 (step S2412).

The display device control unit 801 acquires the high-quality video for the period specified by the transfer start time and the transfer end time transmitted from the camera body 1 in step S2412 (step S2407). That is, the display device control unit 801 (acquisition unit) acquires the edited high-quality video in the camera body 1 in step S2412 by transmitting the editing information to the camera body 1 in step S2406.

The display device control unit 801 next saves the high-quality image acquired in step S2407 in the large-capacity nonvolatile memory 814 (step S2408), and ends this processing.

As described above, according to the second embodiment, after a transfer interval is set by the display device 800 using a small-size low-quality image, a small-size high-quality image consisting of only the transfer interval is transferred to the camera body 1. to the display device 800. As a result, the time required to transfer images from the camera body 1 to the display device 800 can be shortened.

In addition, since a small-sized video is saved, the storage capacity of the display device 800 can be reduced.

(Example 3)
In the second embodiment, the transfer range of the high-quality video is set based on the period specified by the transfer start time and the transfer end time set on the display device. On the other hand, in the third embodiment, the low-quality video is used to edit the image area cut out from the video on the display device, and the transfer range of the high-quality video is set based on the cut-out area (edit information) of the video.

Hereinafter, the present embodiment will be described as being basically derived from the first and second embodiments.

Therefore, in the configuration of the camera system (video transfer system) of Example 3, the same reference numerals are used for the same configuration as the configuration of the camera system of Example 1 or 2, and redundant description is omitted. Details of the configuration will be added and explained each time.

First, the camera body 1 in Example 3 will be described.

FIG. 26 is a functional block diagram of a camera body according to the third embodiment;

The camera body 1 of Example 3 includes a development processing unit 2601, a communication unit 2602, and an image clipping unit instead of the image clipping/development processing unit 50, image recording unit 2301, and communication unit 2304 in the camera body 1 of Example 2. 2603 and an image recording unit 2604 .

The development processing unit 2601 is similar to the image clipping/development processing unit 50, but does not clip the image, and develops the entire image received from the photographing unit 40. FIG.

The communication unit 2602 is similar to the transmission unit 70 , but in addition to transmitting video, it can also receive a clipped region (editing information) of the video from the display device 800 . Edit information received by the communication unit 2602 is passed to the image clipping unit 2603 .

The image clipping unit 2603 (area clipping means) clips a predetermined area from the high-quality video based on the editing information passed from the communication unit 2602 . The video clipped by the image clipping unit 2603 is transmitted to the display device 800 using the communication unit 2602 .

The image recording unit 2604 is similar to the image recording unit 2301 in the second embodiment, but records video in the format shown in FIG. 15 described in the first embodiment. That is, in addition to the video data, the clipped region of the video is recorded as metadata. The clipping position recorded as metadata by the image recording unit 2604 is the spatial clipping position determined by the recording direction/angle of view determining unit 30 .

Next, a cutout area edit screen displayed on the display unit 803 of the display device 800 according to the third embodiment will be described with reference to FIG. 28 .

The user can use the clipping area edit screen to set the image area to be clipped from the high-quality video as the video transferred from the camera body 1 to the display device 800 .

FIG. 28A is a schematic diagram showing an example of the initial state of the cutout area edit screen according to the third embodiment displayed on the display unit 803 of the display device 800. FIG.

As shown in FIG. 28(a), in the initial state of the clipping area edit screen, a message 2801, a video playback section 2802, a rewind button 2803, a play/pause button 2804, a fast forward button 2805, and a transfer start button 2806 are displayed. Is displayed.

The image reproduction unit 2802 reproduces and displays the low image quality image received from the camera body 1 . The user can use the rewind button 2803 , play/pause button 2804 , and fast forward button 2805 to play, pause, and change the playback position of the low-quality video in the video playback section 2802 .

In the example shown in FIG. 28A, the low-quality video being played back by the video playback unit 2802 includes a person 2810 and a person 2811, and as an initial setting, a clipping frame 2812 is drawn around the person 2810 with a dotted line. is superimposed.

The clipping frame 2812 is drawn based on the clipping position attached to the video received from the camera body 1 as metadata. That is, the position of the clipping frame 2812 is the direction the user was looking at when the image was captured.

The position of this clipping frame 2812 can be changed by the user using the clipping area editing screen. Specifically, the user can reset the clipping area by moving the clipping frame 2812 superimposed on the video playback unit 2802 .

In FIG. 28A, as a message 2801, a message is displayed notifying that the clipping area can be edited by moving a clipping frame 2812 represented by a dotted line.

For example, when the user was looking at the person 2810 at the time of shooting, but wants to keep the image of the person 2811 as a high-quality image, the user can set the clipping frame 2812 superimposed and displayed on the image playback unit 2802 to the person 2811 . move to a position surrounding

FIG. 28(b) is a schematic diagram showing the state of the clipping area editing screen when the clipping frame 2812 is moved to a position surrounding the person 2811 on the clipping area editing screen shown in FIG. 28(a).

Although the size of the clipping frame 2812 is fixed in the third embodiment, the size may also be changeable.

Next, video transfer processing of the camera system according to the third embodiment will be described.

FIG. 27 is a flowchart of video transfer processing of the camera system according to the third embodiment.

For ease of explanation, in this flowchart, the processing in the display device 800 is shown on the left side, and the processing by the camera body 1 is shown on the right side.

In other words, each process of steps S2701 to S2707 is realized in the display device 800 by the display device control section 801 executing the program stored in the built-in non-volatile memory 812. FIG. On the other hand, each process of steps S2709 to S2712 is realized in the camera body 1 by the overall control CPU 101 executing a program stored in the built-in non-volatile memory 102. FIG.

Also, this flowchart starts from a state in which the display device 800 and the camera body 1 are already connected by high-speed wireless.

When the display device control unit 801 receives a video transfer request from the camera body 1 , first, the display device control unit 801 transmits a low-quality video acquisition request to the camera body 1 . As a result, the display device control section 801 uses the high-speed wireless unit 872 or the low-power wireless unit 871 to obtain a low-quality image from the camera body 1 (step S2701). When the acquisition of the low-quality video is completed, the display device control unit 801 proceeds to step S2702.

Overall control CPU 101 uses communication unit 2602 to transmit the low-quality video to display device 800 in response to the acquisition request transmitted from display device 800 in step S2701 (step S2709). The video transmitted in step S2709 is read from the image recording unit 2604, and is an ultra-wide-angle video before clipping processing, to which metadata regarding the clipping position has been added.

In step S2702, the display device control unit 801 displays a clipping area edit screen, and then advances to step S2703 to determine whether or not the position of the clipping frame 2812 has been moved from the default position by the user.

If the position of the clipping frame 2812 has been moved from the default position (YES in step S2703), the destination of the clipping frame 2812 is stored as a new clipping area in step S2704, and the flow advances to step S2705. On the other hand, if the position of the clipping frame 2812 has not been moved from the default position (NO in step S2703), the process proceeds directly to step S2705.

In step S2705, the display device control unit 801 determines whether or not the user has pressed the transfer start button 2806 to instruct the start of transfer.

If no transfer start instruction has been issued (NO in step S2705), the display device control unit 801 returns to step S2703 to continue displaying the cutout area editing screen. On the other hand, if a transfer start instruction has been issued (YES in step S2705), the process advances to step S2706, and the display device control unit 801 transmits the cutout area to the camera body 1 as editing information.

Overall control CPU 101 uses communication unit 2602 to receive the clipping area (edit information) transmitted from display device 800 in step S2706 (step S2710).

Overall control CPU 101 then cuts out a high-quality image based on the editing information received in step S2710 (step S2711). That is, an image of a predetermined position and a predetermined size (predetermined area), that is, only the image in the cropping area is cut out from the high-quality ultra-wide-angle image, thereby generating a high-quality but small-sized image.

Overall control CPU 101 then transmits the high-quality video clipped in step S2711 to display device 800 (step S2712).

The display device control unit 801 acquires a high-quality image obtained by clipping only the image in the clipping area from the high-quality ultra-wide-angle image transmitted from the camera body 1 in step S2712 (step S2707).

The display device control unit 801 next saves the high-quality image acquired in step S2707 in the large-capacity nonvolatile memory 814 (step S2708), and ends this process.

As described above, according to the third embodiment, after a clipping region is set by the display device 800 using a small low-quality video, a small high-quality video consisting of only the clipping region is generated by the camera body 1. to the display device 800. As a result, the time required to transfer images from the camera body 1 to the display device 800 can be shortened.

Also, as in the second embodiment, since a small-sized video is saved, the storage capacity of the display device 800 can also be reduced.

Further, by combining the temporal video clipping described in the second embodiment and the regional video clipping described in the third embodiment, it is possible to further reduce the transfer time and the recording capacity.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

1 camera body 10 photographing/detecting unit 11 power switch 12 imaging mode switch 13 face direction detection window 14 start switch 15 stop switch 16 imaging lens 17 LED
18 chest connection pad 19L, 19R microphone 20 face direction detection unit 30 recording direction/angle of view determination unit 40 photographing unit 50 image cutting/development processing unit 60 primary recording unit 70 transmission unit 80 connection unit 81 angle holding unit 82 band unit 83 connection Surface 84 Electric Cable 90 Battery Part 91 Charging Cable Insertion Port 92L, 92R Adjustment Button 93 Backbone Protection Notch 94 Battery 101 Overall Control CPU
102 built-in nonvolatile memory 103 primary memory 105 speaker 106 vibrator 107 angular velocity sensor 108 acceleration sensor 805 in-camera 806 face sensor 800 display device 801 display device control unit 807 angular velocity sensor 808 acceleration sensor 809 imaging signal processing circuit 850 calibrator 900 simple display device 901 action camera 902 head fixed accessory 903 omnidirectional imaging camera 904 lens 905 shooting button 2301, 2604 image recording unit 2302 image quality conversion unit 2304, 2602 communication unit 2601 development processing unit 2603 image clipping unit

Claims (18)

A video transfer system comprising an imaging device and a display device,
The imaging device is
imaging means;
a conversion means for converting the first image captured by the imaging means into a second image having a smaller data size than the first image;
recording means for recording the first image and the second image;
a first means of communication;
When the editing information is received from the display device by the first communication means, the first video is edited based on the received editing information, and the edited first video is transmitted by the first communication means. and image editing means for transmitting to a display device,
The display device
a second means of communication;
editing information setting means for receiving the second image from the imaging device by the second communication means and setting the editing information based on the received second image;
acquisition means for acquiring the edited first image from the imaging device through the second communication means by transmitting the editing information to the imaging device through the second communication means; video transmission system. 2. The video transfer system according to claim 1, wherein said edit information setting means sets a temporal cut-out section as the edit information. 3. The video transfer system according to claim 2, wherein said image editing means cuts out a predetermined section of said first video based on said temporal cut-out section set as editing information. The display device
extraction means for dividing the received second video by a predetermined period and extracting a representative image representing each divided video;
4. The video transfer system according to claim 3, further comprising display means for arranging and displaying representative images representative of the divided images in chronological order. The editing information setting means uses two images selected by the user from among the representative images displayed on the display means to set the clipping section of the temporal image. Item 5. The video transfer system according to item 4. 2. The video transfer system according to claim 1, wherein said edit information setting means sets an image clipping area as the edit information. 7. The image transfer system according to claim 6, wherein said image editing means is area clipping means for clipping a predetermined area from said first image based on the image clipping area set as said editing information. . The imaging device is
face orientation detection means for detecting the face orientation of the user;
8. The video transfer system according to claim 7, further comprising clipping area determining means for determining a clipping area of said image based on the user's face direction detected by said face direction detecting means. The area clipping means clips a predetermined area from the first image based on the clipping area determined by the clipping area determining means until the editing information is received by the second communication means;
the conversion means converts the first image cut out by the area cutout means to generate the second image;
9. The video transfer system according to claim 8, wherein said first video is stored in said recording means after being clipped by said region clipping means. 9. The video transfer system according to claim 8, wherein said recording means records the cutout area determined by said cutout area determination means together with said second image. 11. The video transfer system according to claim 10, wherein said display device further comprises display means for superimposing and displaying the clipping region determined by said clipping region determination device on said second video. 12. The editing information setting device according to claim 11, wherein, when the clipped region displayed on the display device is moved by a user, the edited clipping region is set in the editing information. video transmission system. An imaging device that constitutes a video transfer system together with a display device,
imaging means;
a conversion means for converting the first image captured by the imaging means into a second image having a smaller data size than the first image;
recording means for recording the first image and the second image;
a first means of communication;
When the editing information is received from the display device by the first communication means, the first video is edited based on the received editing information, and the edited first video is transmitted by the first communication means. An imaging device, comprising: image editing means for transmitting to the display device. A display device that constitutes a video transfer system together with an imaging device that captures a first video,
a second means of communication;
receiving a second image having a data size smaller than that of the first image converted from the first image from the imaging device by the second communication means, and editing based on the received second image editing information setting means for setting information;
acquisition means for acquiring the edited first image from the imaging device through the second communication means by transmitting the editing information to the imaging device through the second communication means. display device. A control method for an imaging device forming a video transfer system together with a display device,
The imaging device is
imaging means;
a conversion means for converting the first image captured by the imaging means into a second image having a smaller data size than the first image;
recording means for recording the first image and the second image;
a first means of communication;
a receiving step of receiving edit information from the display device by the first communication means;
an editing step of editing the first video based on the received editing information;
and a transmission step of transmitting the edited first image to the display device through the first communication means. A control method for a display device that constitutes a video transfer system together with an imaging device that captures a first video,
The display device has a second communication means,
a receiving step of receiving a second image having a data size smaller than that of the first image converted from the first image from the imaging device by the second communication means;
an editing information setting step of setting editing information based on the received second video;
and an obtaining step of obtaining the edited first image from the imaging device by the second communication means by transmitting the editing information to the imaging device by the second communication means. control method. 14. A computer-executable program that causes a computer to function as steps of the imaging device according to claim 13. A computer-executable program that causes a computer to function as steps of a display device according to claim 14.

Images