JP2021071505A - Focal point adjustment support device and program - Google Patents

Abstract

To appropriately support the adjustment of a focus point by an imaging person.SOLUTION: A focal point adjustment supporting device for supporting the adjustment by an imaging person of the focus point of a first camera includes: a synthesis unit for synthesizing a first image of a first object taken by a first camera of which focus position is adjusted in response to an operation by the imaging person, and synthesizing a second image, which is displaced more from the position of the first object as the distance from the focus position of the first camera to the first object becomes longer; and a display control unit for displaying a third image synthesized by the synthesis unit in a screen.SELECTED DRAWING: Figure 3

Description

The present invention relates to a focus adjustment assisting device and a program.

Conventionally, in a camera that captures images such as moving images and still images, there is known a technique that enables the photographer to focus (focus, focus) by a manual operation such as turning the focus ring. The manual adjustment of the focus position (focus position) of the camera by the photographer is also called manual focus or the like. When the photographer manually focuses, the photographer can operate the focus ring to take an image in focus on the subject so that the outline of the subject reflected in the viewfinder is minimized.

Patent No. 5723489 Patent No. 6071748 Japanese Unexamined Patent Publication No. 2014-103490

In the prior art, the photographer may not be able to focus properly. The disclosed technique is intended to provide a technique capable of appropriately assisting the photographer in adjusting the focusing position.

The first camera according to an embodiment of the present disclosure, wherein the focus adjustment assisting device that assists the photographer in adjusting the focusing position of the first camera adjusts the focusing position in response to an operation by the photographer. Combines the first image in which the first object is photographed and the second image in which the amount of deviation in the position of the first object increases as the distance from the in-focus position of the first camera to the first object increases. It has a compositing unit and a display control unit for displaying a third image synthesized by the compositing unit on a screen.

According to the disclosed technique, it is possible to appropriately assist the photographer in adjusting the focusing position.

It is a figure which shows the configuration example of the image capturing system which concerns on embodiment. It is a functional block diagram which shows the functional structure of the control device which concerns on embodiment. It is a sequence diagram which shows an example of the processing of the image taking system which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment. It is a figure explaining an example of the focus confirmation image which concerns on embodiment.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

<System configuration>
FIG. 1 is a diagram showing a configuration example of the image capturing system 1 according to the embodiment. In FIG. 1, the image capturing system 1 includes a photographing device 2 and a focus adjustment assisting device 3. The image capturing system 1 is provided by the photographing device 2 in which the position where the camera is focused (focused, focused) (hereinafter, also appropriately referred to as “focus position”) is adjusted by the photographer using the focus adjustment assisting device 3. This is a system for capturing images such as moving images and still images. The photographing device 2 and the focus adjustment assisting device 3 may be integrated. In this case, the photographing device 2 and the focus adjustment assisting device 3 may be housed in the same housing.

The image capturing system 1 may be an information processing terminal such as a smartphone having a plurality of cameras and a touch panel.

≪Shooting device 2≫
The photographing device 2 is a device that captures images such as moving images and still images. In the example of FIG. 1, the photographing device 2 includes a camera (camera for photographing, a main camera) 21, a lens 22, and an adjusting device 23. The camera 21 is a device for capturing an image, and has 4K (horizontal 4,000, vertical screen resolution of about 2,000) and 8K (horizontal 8,000, vertical screen resolution of about 4,000) for broadcasting. A digital camera or the like capable of taking an image may be used. The image taken by the camera 21 may be recorded in a built-in or external recording device, or may be output to an external device such as for broadcasting. The lens 22 is a lens used for taking an image with the camera 21.

The adjustment device 23 is a device that adjusts (changes, sets) the focus and zoom of an image captured by the camera 21 in response to a direct or indirect operation by the photographer, and is a focus ring (focus ring). ), And may have a zoom ring and the like.

The adjustment device 23 outputs information indicating the current focus and zoom status to the control device 34 of the focus adjustment auxiliary device 3 in response to a direct operation by the photographer such as turning the focus ring. You may. Further, the adjustment device 23 may adjust the focus and the zoom based on the signal received from the control device 34 of the focus adjustment assisting device 3 by an indirect operation by the photographer.

≪Focus adjustment assist device 3≫
The focus adjustment assisting device 3 is a device that assists the photographer in adjusting the focusing position of the camera 21. The focus adjustment assist device 3 includes a camera 31 (focus adjustment assist camera, sub camera), a lens 32, an adjustment device 33, a control device 34, and a display device 35.

The camera 31 is a device that captures an image from a position different from that of the camera 21, and may be a digital camera or the like. The image taken by the camera 31 is only displayed to the photographer for adjusting the focusing position and is not recorded, so that the resolution is lower than that of the image taken by the camera 21. May be good. The lens 32 is a lens used for taking an image with the camera 31.

The adjusting device 33 is a device that adjusts the focus and zoom of the image captured by the camera 31 based on the signal received from the control device 34. The configuration may not include the adjusting device 33.

The control device 34 is a device that generates an image for focus adjustment based on an image taken by the camera 21 and an image taken by the camera 31, and displays the generated image on the display device 35. The control device 34 may be a computer having a CPU (Central Processing Unit), a memory, an auxiliary storage device such as an SSD (Solid State Drive), an external interface, and the like. In this case, one or more programs may be installed via the external interface.

The display device 35 is a device (finder, view finder) that displays an image for focus adjustment output by the control device 34, and is a liquid crystal display (LCD, liquid crystal display), a cathode ray tube (CRT, Cathode Ray Tube). , An organic electroluminescence display (OLED, Organic Light Emitting Display), a plasma display (PDP, Plasma Display Panel), or the like may be used. Further, the display device 35 may have a touch panel. The display device 35 may be included in the photographing device 2.

<Functional configuration of control device 34>
Next, with reference to FIG. 2, the functional configuration of the control device 34 according to the embodiment will be described. FIG. 2 is a functional block diagram showing a functional configuration of the control device 34 according to the embodiment.

The control device 34 includes a determination unit 341, a generation unit 342, a synthesis unit 343, a display control unit 344, a reception unit 345, and an adjustment unit 346. Each of these parts is realized by the cooperation of one or more programs installed in the control device 34 and the CPU or the like of the control device 34.

The determination unit 341 determines the focusing position of the camera 21, the zoom, and the like.

The generation unit 342 generates a second image in which the position of the first object is shifted according to the distance from the in-focus position of the camera 21 to the first object and the distance from the camera 21 to the first object.

The compositing unit 343 synthesizes the first image in which the first object is photographed by the camera 21 whose focusing position is adjusted in response to the operation of the photographer and the second image generated by the generating unit 342.

The display control unit 344 displays the image synthesized by the synthesis unit 343 on the screen of the display device 35.

The reception unit 345 receives various operations from the photographer. The reception unit 345 receives an operation from the photographer to adjust (change, set) the focusing position and zoom of the camera 21.

The adjustment unit 346 adjusts the focusing position and zoom of the camera 21. The adjusting unit 346 may transmit a control signal to the adjusting device 23 of the photographing device 2 in response to the operation of the photographer received by the receiving unit 345. The adjusting device 23 of the photographing device 2 may be directly operated by the photographer.

<Processing of image shooting system>
Next, the processing of the image capturing system 1 according to the embodiment will be described with reference to FIGS. 3 to 7C. FIG. 3 is a sequence diagram showing an example of processing of the image capturing system 1 according to the embodiment. 4A to 7C are diagrams illustrating an example of a focus confirmation image according to the embodiment.

In step S1, the determination unit 341 determines the current focusing position and zoom of the camera 21. Here, the determination unit 341 may acquire information on the current focusing position and zoom of the camera 21 from the adjustment device 23 of the photographing device 2. In this case, the adjusting device 23 of the photographing device 2 may have a sensor for measuring the focusing position and the zoom of the camera 21 by the lens 22. The sensor may be a sensor mechanically, electronically or optically coupled to a focusing position adjusting mechanism (for example, a focus ring) of the lens 22 and a zoom adjusting mechanism (for example, a zoom ring). More specifically, the sensor may be a focus ring, a rotary encoder or a potentiometer mechanically interlocked with the zoom ring, or the like.

The sensor may output the detected value to the determination unit 341 as it is. Alternatively, the sensor may convert the detected value into information indicating the in-focus position and the zoom and output it to the determination unit 341. In this case, the sensor may use a pre-calibrated look-up table to convert the detected values into information indicating the focus position and zoom. The focusing position (depth distance Z _P ) may be the distance from the focal plane of the camera 21 to the focusing position, or the distance from the first principal point of the camera 21 to the focusing position. The information indicating the zoom may include a magnification value.

The determination unit 341 may acquire information on the current focusing position and zoom of the camera 21 from the adjustment unit 346.

Subsequently, in step S2, the generation unit 342 generates a composite image (an example of the "second image") based on the focus and zoom of the camera 21. Here, the generation unit 342 shifts the position of the subject according to the distance from the in-focus position of the camera 21 to the subject (an example of the "first object") and the distance from the camera 21 to the subject. Generate a composite image.

The generation unit 342 may generate a composite image in which the amount of deviation of the position of the subject increases as the distance from the in-focus position of the camera 21 to the subject increases. As a result, when the image of the camera 21 and the image for compositing are combined, the photographer can focus on the combined image in order to focus on the subject as the amount of deviation of a certain subject increases. It is possible to understand that it is necessary to increase the amount of position adjustment.

Further, the generation unit 342 may generate an image in which the amount of deviation in the position of the subject is reduced as the distance from the camera 21 to the subject increases. As a result, when the image of the camera 21 and the image for compositing are combined, the photographer can easily adjust the focusing position as the subject is closer because the amount of deviation of the subject is larger as the subject is closer. Can be done.

The generation unit 342 recognizes (extracts, detects) each subject from the image of the camera 21 by image recognition, calculates the distance from the camera 21 to each subject, and based on each calculated distance, the position of each subject. You may generate a composite image in which the above is shifted. In this case, the generation unit 342 may calculate the parallax of each subject from the stereo camera images taken by the camera 21 and the camera 31, and calculate each distance from the camera 21 to each subject based on the calculated parallax. Alternatively, the generation unit 342 may calculate each distance from the camera 21 to each subject by a sensor such as a lidar (LIDAR: Light Detection and Ranging, Laser Imaging Detection and Ranging).

The generation unit 342 may generate a composite image in which the images taken by the cameras 31 installed side by side on the camera 21 are shifted according to the focusing position of the camera 21. As a result, the greater the distance from the in-focus position of the camera 21 to the subject, the greater the amount of deviation in the position of the subject, and the greater the distance from the camera 21 to the subject, the greater the amount of deviation in the position of the subject. It is possible to generate a small composite image.

<< Example of image generation processing for compositing >>
In step S2, an example of processing of the generation unit 342 when generating a composite image in which the images taken by the cameras 31 installed side by side on the camera 21 are shifted according to the focusing position of the camera 21 will be described. ..

The relative position and the relative posture of the camera 31 with respect to the camera 21 are arbitrary. In the example of FIG. 1, the optical axis of the lens 22 and the optical axis of the lens 32 are parallel, and the straight line connecting the first main point P of the lens 22 and the first main point Q of the lens 32 is the optical axis of the lens 22. , And the optical axis of the lens 32, respectively. Let B be the length of the line segment PQ.

With the first main point P of the lens 22 as the origin, the X-axis is in the direction from the point P to the first main point Q of the lens 32, the optical axis of the main lens is the Z-axis (the subject direction is positive), and the X-axis- _{Let the Cartesian coordinate system Σ C} be the Cartesian coordinate system with the Y axis in the direction of forming the right-handed system in the order of Y axis-Z axis.

The image captured by the camera 21 is defined as the image I, and the pixel value at the image coordinates (x, y) is defined as I (x, y). The image coordinate system of the camera 21 is Σ _P , the x-axis is in the same direction as the X-axis, and the y-axis is in the same direction as the Y-axis. Further, the focal length of the lens 22 is f _P.

Further, the image captured by the camera 31 is defined as the image J, and the pixel value at the image coordinates (s, t) is defined as J (s, t). The image coordinate system of the camera 31 is Σ _Q , the s axis is in the same direction as the X axis, and the t axis is in the same direction as the Y axis. _{Further, let f Q be} the focal length of the lens of the camera 31.

When the lens 22 is applied to the pinhole model, the image [x ^{y] T} of the image I of the coordinate in the camera coordinate system sigma _C (X, Y, Z) is given by the following expression (1).

また、カメラ３１のレンズにピンホールモデルを適用すれば、カメラ座標系Σ_Ｃにおいて座標（Ｘ,Ｙ，Ｚ）の画像Ｊ上の像［ｘ ｙ］^Ｔは、以下の式（２）となる。

Further, by applying the lens to the pinhole model of a camera 31, an image [x y] ^T of the image J of coordinates in the camera coordinate system sigma _C (X, Y, Z) is given by the following expression (2) ..

上記式（１）、及び式（２）より、以下の式（３）が得られる。

From the above equations (1) and (2), the following equation (3) can be obtained.

式（３）により、奥行き距離Ｚの位置にある点が投影された画像Ｉ上の点（ｘ,ｙ）は、画像Ｊ上において点（ｆｑ・ｘ／ｆｐ−ｆｑ・ｘ／Ｚ, ｆｑ・ｙ／ｆｐ）に対応することが示されている。したがって、画像Ｊにおいては、画像Ｉを（ｆｑ／ｆｐ）倍したものが、奥行き距離Ｚが小さいほどｓ軸の負方向にずれて観測される。また、奥行き距離Ｚが無限遠に近づくほど、ずれは０に漸近する。

According to the equation (3), the points (x, y) on the image I on which the points at the position of the depth distance Z are projected are the points (fq · x / fp-fq · x / Z, fq · · on the image J. It has been shown to correspond to y / fp). Therefore, in the image J, the image I multiplied by (fq / fp) is observed as being deviated in the negative direction of the s axis as the depth distance Z is smaller. Further, as the depth distance Z approaches infinity, the deviation gradually approaches zero.

The generation unit 342 magnifies the image J of the camera 31 both vertically and horizontally (fp / fq) times, and only (fq · B) / Z in the s-axis direction according to the _{focusing position Z P determined by the determination unit 341.} A translated image for compositing (shifted image) K (Z _P ) (an example of a "second image") may be output. _{In this case, the pixel value K (Z P} ; x, y) at the image coordinates (x, y) of the composite image K is given by the following equation (4). As a result, a composite image shifted according to the focusing position of the camera 21 is generated.

なお、式（４）はピンホールモデルを仮定しているが、光学系によっては合焦位置の調整に伴って像が拡大または縮小することや、ひずみ（例えば、樽型や糸巻き型のひずみ）が生じることやひずみが変化することが生じ得る。そのため、生成部３４２は、画像Ｉと画像Ｊとの間の倍率やひずみの度合いが整合するよう、判定部３４１により判定された合焦位置Ｚ_Ｐに応じて幾何補正を行うようにしてもよい。

Equation (4) assumes a pinhole model, but depending on the optical system, the image may be enlarged or reduced as the focusing position is adjusted, or strain (for example, barrel-shaped or pincushion-shaped strain). Can occur or the strain can change. Therefore, the generation unit 342 may perform geometric correction according to the _{focusing position Z P} determined by the determination unit 341 so that the magnification and the degree of distortion between the image I and the image J are matched. ..

Subsequently, in step S3, the compositing unit 343 confirms the focus by synthesizing the image taken by the camera 21 (an example of the “first image”) and the compositing image generated by the generating unit 342. An image for use (an example of a "third image") is generated.

In the following, for each subject located closer to the in-focus position of the camera 21 when viewed from the camera 21, the generation unit 342 is largely shifted to the left as the distance from the camera 21 is closer, and is farther than the in-focus position. For each subject located in, an example will be described in which a composite image is generated that is shifted to the right as the distance from the camera 21 increases. The direction in which the generation unit 342 shifts the position of each subject is not limited to the horizontal direction. The synthesis unit 343 may be shifted in the vertical direction or the diagonal direction.

Below, in the example of FIGS. 4A to 7C, each subject such as the soccer ball 502, the first player 503, the second player 504, and the background mountain 505 is captured in the image taken by the camera 21. It will be explained as a thing.

Further, in the examples of FIGS. 4A, 5A, 6A, and 7A, the composite image when the focusing position of the camera 21 is close to infinity includes each subject included in the image taken by the camera 21. It is assumed that each subject such as a soccer ball 512, a first player 513, a second player 514, and a mountain 515 in the background, which are displaced from each other, is photographed.

Further, in the examples of FIGS. 4B, 5B, 6B, and 7B, the composite image when the focusing position of the camera 21 is adjusted to the position of the second player 504 was taken by the camera 21. It is assumed that each subject included in the image is captured, such as a soccer ball 522, a player 523, a player 524, and a mountain 525 in the background, in which the positions of the subjects are shifted.

Further, in the examples of FIGS. 4C, 5C, 6C, and 7C, the composite image when the focusing position of the camera 21 is adjusted to the position of the first player 503 was taken by the camera 21. It is assumed that each subject included in the image is captured, such as a soccer ball 522, a player 523, a player 524, and a mountain 525 in the background, in which the positions of the subjects are shifted.

≪Example of arranging≫
As shown in FIGS. 4A to 4C, the compositing unit 343 synthesizes the image taken by the camera 21 and the compositing image, and the image taken by the camera 21 and the compositing image are arranged for focus confirmation. An image may be generated.

In the example of FIGS. 4A to 4C, in the focus confirmation image 400, the image taken by the camera 21 is displayed in the upper area 401, and the composite image is displayed side by side with the image taken by the camera 21 on the lower side. ing.

FIG. 4A shows an example of a focus confirmation image when the focusing position of the camera 21 is close to infinity. The horizontal distance (deviation) 424 between the mountains 505 and 515 in the background where the distance of the camera 21 is close to infinity is almost 0 (none). Further, the deviation 421 between the soccer balls 502 and 512, which are the closest to the camera 21, the deviation 422 between the next closest first players 503 and 513, and the deviation 423 between the next closest second players 504 and 514. In order, the deviation becomes smaller.

FIG. 4B shows an example of a focus confirmation image when the focusing position of the camera 21 is adjusted to the position of the second player 504. Therefore, in FIG. 4B, the deviation 443 between the second player 504 and 524 existing in the in-focus position is 0. Further, among the subjects located closer to the in-focus position of the camera 21 when viewed from the camera 21, the deviation 441 between the soccer ball 502 and 522, which are the closest to the camera 21, and the next closest first player 503. The deviation to the left side becomes smaller in the order of the deviation 442 from that of 523 and 523. Further, the deviation 444 between the background peaks 505 and 525, which are located farther than the in-focus position of the camera 21 when viewed from the camera 21, is a deviation to the right.

FIG. 4C shows an example of a focus confirmation image when the focusing position of the camera 21 is adjusted to the position of the first player 503. Therefore, in FIG. 4C, the deviation 462 between the first player 503 and 533 existing in the in-focus position is 0. Further, the deviation 461 between the soccer ball 502 and 535, which are located closer to the in-focus position of the camera 21 when viewed from the camera 21, is a deviation to the left. In addition, among the subjects located farther than the in-focus position of the camera 21 when viewed from the camera 21, the deviation 463 between the second player 504 and 534, which are the closest to the camera 21, and the mountain in the background closest to the second player 504. The deviation to the right increases in the order of the deviation 464 between 505 and 535.

The compositing unit 343 may switch from FIG. 4A to a split screen as shown in FIG. 4C when the “arrangement mode” is specified by the photographer's setting operation.

≪Example of superimposition≫
As shown in FIGS. 5A to 5C, the compositing unit 343 synthesizes the image taken by the camera 21 and the compositing image, and the image taken by the camera 21 and the compositing image are superimposed for the focus confirmation. An image may be generated.

In this case, the compositing unit 343 combines the image I and the compositing image K (Z _P ) at a ratio (1-α): α in the superposed region as shown in the following equation (5). It may be output as a focus confirmation image L (Z _P). In addition, α is a real number from 0 to 1.

なお、合成部３４３は、撮影者の設定操作によりαの値を変更できるようにしてもよい。この場合、プッシュスイッチ、またはトグルスイッチ等のスイッチの操作により、αの値を０、０．５、及び１のように離散的な値に設定できるようにしてもよい。または、ボリュームやロータリエンコーダなどによって連続的または多値で設定できるようにしてもよい。

The compositing unit 343 may be able to change the value of α by the setting operation of the photographer. In this case, the value of α may be set to discrete values such as 0, 0.5, and 1 by operating a switch such as a push switch or a toggle switch. Alternatively, it may be set continuously or with multiple values by a volume, a rotary encoder, or the like.

FIG. 5A shows an example of a focus confirmation image when the focusing position of the camera 21 is close to infinity. The horizontal distance (deviation) between the ridges 505 and 515 in the background where the distance of the camera 21 is close to infinity is almost 0 (none). Further, as the distance from the camera 21 is shorter, each subject in the composite image is shifted to the left side more than the position of each subject in the image taken by the camera 21. Therefore, the deviation between the soccer balls 502 and 512, which are the closest to the camera 21, the deviation between the first players 503 and 513, which are the next closest, and the deviation between the second players 504 and 514, which are the next closest, in that order. Is getting smaller.

FIG. 5B shows an example of a focus confirmation image when the focusing position of the camera 21 is adjusted to the position of the second player 504. Therefore, in FIG. 5B, the deviation between the second player 504 and 524 existing in the in-focus position is 0.

Therefore, among the subjects located closer to the in-focus position of the camera 21 when viewed from the camera 21, the deviation between the soccer balls 502 and 522, which are the closest to the camera 21, and the first player 503, which is the next closest. The deviation to the left decreases in the order of deviation from 523. Further, the deviation between the background peaks 505 and 525, which are located farther than the in-focus position of the camera 21 when viewed from the camera 21, is a deviation to the right.

FIG. 5C shows an example of a focus confirmation image when the focusing position of the camera 21 is adjusted to the position of the first player 503. Therefore, in FIG. 5C, the deviation between the first player 503 and 533 existing in the in-focus position is zero. Further, the deviation between the soccer ball 502 and 532, which are located closer to the in-focus position of the camera 21 when viewed from the camera 21, is a deviation to the left. Further, among the subjects located farther than the in-focus position of the camera 21 when viewed from the camera 21, the deviation between the second player 504 and 534, which are the closest to the camera 21, and the next closest background mountain 505. The deviation to the right increases in the order of the deviation between and 535.

The compositing unit 343 may switch from FIG. 5A to the superimposed screen as shown in FIG. 5C when the “superimposed mode” is specified by the setting operation of the photographer.

(Superimposition according to the intelligibility of the contour)
The compositing unit 343 may superimpose the region of each subject included in the compositing image with a transmittance corresponding to the clarity of the outline of each subject captured in the image captured by the camera 21. As a result, in a region where the outline of the subject is likely to be unclear, it is difficult to assist in adjusting the focusing position even if the images are superimposed, and the image is only unsightly. Therefore, it is possible to prevent the images from being superimposed. Therefore, the visibility of the image is less likely to be hindered, and the photographer can concentrate on the area where it is easy to focus.

In this case, the compositing unit 343 divides the image captured by the camera 21 into a plurality of regions. Then, the larger the total value of the brightness (brightness) of the pixels included in the region, the closer the transmittance of the region corresponding to the region in the composite image is to a predetermined value (for example, 0.5). The smaller the total value of brightness and the like, the closer the transmittance may be to zero.

Further, the compositing unit 343 sets the transmittance of the region corresponding to the region in the compositing image to a value closer to a predetermined value (for example, 0.5) as the contrast of the image included in the region is larger, and the contrast or the like is set. The smaller the value, the closer the transmittance may be to 0.

Further, the compositing unit 343 recognizes the contour region of the subject by image recognition, and the greater the degree of edge-likeness of the recognized region, the more the transmittance of the region corresponding to the region in the compositing image is set to a predetermined value ( For example, the value may be set closer to 0.5), and the smaller the degree of edge-likeness, the closer the transmittance may be to 0.

(Superimposition of difference images)
The compositing unit 343 may superimpose an image based on the difference between the value of each pixel of the image taken by the camera 21 and the value of each pixel of the compositing image on the image taken by the camera 21. As a result, only the vicinity of the contour of the subject is superimposed, so that the visibility of the image is not hindered, and the photographer can concentrate on the area where the focus is easy to focus.

In this case, the compositing unit 343 calculates the absolute value of the difference between the value of each pixel of the image captured by the camera 21 and the value of each pixel of the compositing image. Then, the image of the absolute value of the calculated value of each pixel may be superimposed on the image taken by the camera 21 as a composite image. As a result, the pixels in which the pixel values of the image captured by the camera 21 and the pixel values of the composite image match can be prevented from being superimposed on the image captured by the camera 21.

Further, the compositing unit 343 calculates the absolute value of the difference between the value of each pixel of the image captured by the camera 21 and the value of each pixel of the compositing image. Then, the image of the absolute value of the calculated value of each pixel may be output as the focus confirmation image. In this case, the subject located at the in-focus position is displayed in black, and the area of the subject far from the in-focus position is displayed in a color other than black.

≪Example of arranging by replacing some parts≫
The compositing unit 343 may generate a focus confirmation image in which a part of the region of the image captured by the camera 21 is replaced with a part of the region of the compositing image. In the example of FIGS. 6A to 6C, the synthesis unit 343 is divided into the first region and the third region among the first region, the second region (central region), and the third region divided horizontally from the top. An image of each region included in the image taken by the camera 21 is set, and a focus confirmation image in which the image of the region included in the composite image is set is generated in the second region. The compositing unit 343 may be able to change the vertical position and height of the second region by the setting operation of the photographer.

In the example of FIGS. 6A to 6C, similarly to the example of FIGS. 5A to 5C, the images taken by the camera 21 include a soccer ball 502, a first player 503, a second player 504, a background mountain 505, and the like. The subject is photographed.

In the example of FIGS. 6A to 6C, the image captured by the camera 21 is combined with the first region 602 and the third region 604, and the composite image is combined with the second region 603. It differs from the example of FIGS. 5A to 5C.

FIG. 6A shows an example of the focus confirmation image 601 when the focusing position of the camera 21 is close to infinity. FIG. 6B shows an example of the focus confirmation image 611 when the focusing position of the camera 21 is adjusted to the position of the second player 504. FIG. 6C shows an example of the focus confirmation image 621 when the focusing position of the camera 21 is adjusted to the position of the first player 503.

The amount and direction of deviation of each subject in the composite image in FIGS. 6A, 6B, and 6C are the same as in the cases of FIGS. 5A, 5B, and 5C, respectively.

In the example of FIGS. 6A to 6C, the focus confirmation image L (Z _P ) may be formulated as the following equation (6).

なお、β及びγは、０≦β≦γ＜Ｈ（Ｈは焦点確認用画像Ｌ(Ｚ_Ｐ)の垂直画素数）となる実数（典型的には整数）である。β及びγは定数としてもよいし、撮影者の設定操作により変更できるようにしてもよい。また、合成部３４３は、撮影者の設定操作により「一部を置換して並べるモード」が指定された場合に、Ｃ＝１と設定し、以下の式（７）のように図６Ａから図６Ｃのような３分割画面に切り替えてもよい。

Note that β and γ are real numbers (typically integers) such that 0 ≦ β ≦ γ <H (H is the _{number of vertical pixels of the focus confirmation image L (Z P)).} β and γ may be constants, or may be changed by the photographer's setting operation. Further, the compositing unit 343 sets C = 1 when the "mode of arranging by replacing a part" is specified by the setting operation of the photographer, and is shown from FIG. 6A as shown in the following equation (7). You may switch to a 3-split screen such as 6C.

≪一部を重畳して並べる例≫
合成部３４３は、カメラ２１で撮影された画像の領域の一部に、合成用画像の領域の一部が重畳された焦点確認用画像を生成してもよい。図７Ａから図７Ｃの例では、合成部３４３は、水平方向に上から分割された第１領域、第２領域、及び第３領域のうち、第１領域、及び第３領域に、カメラ２１で撮影された画像に含まれる各領域の画像が設定され、第２領域に、カメラ２１で撮影された画像に含まれる領域の画像と合成用画像に含まれる領域の画像とが重畳された焦点確認用画像を生成している。

≪Example of superimposing and arranging some parts≫
The compositing unit 343 may generate a focus confirmation image in which a part of the region of the compositing image is superimposed on a part of the region of the image captured by the camera 21. In the example of FIGS. 7A to 7C, the compositing unit 343 is divided into the first region and the third region among the first region, the second region, and the third region horizontally divided from the top by the camera 21. The image of each region included in the captured image is set, and the focus confirmation in which the image of the region included in the image captured by the camera 21 and the image of the region included in the composite image are superimposed on the second region. Image is being generated.

In the example of FIGS. 7A to 7C, similarly to the example of FIGS. 5A to 5C, the images taken by the camera 21 include a soccer ball 502, a first player 503, a second player 504, a background mountain 505, and the like. The subject is photographed.

In the example of FIGS. 7 to 7C, the second region 603 is different from the example of FIGS. 5A to 5C in that the image captured by the camera 21 and the composite image are superimposed.

FIG. 7A shows an example of the focus confirmation image 701 when the focusing position of the camera 21 is close to infinity. FIG. 7B shows an example of the focus confirmation image 711 when the focusing position of the camera 21 is adjusted to the position of the second player 504. FIG. 7C shows an example of the focus confirmation image 721 when the focusing position of the camera 21 is adjusted to the position of the first player 503.

The amount and direction of deviation of each subject in the composite image in FIGS. 7A, 7B, and 7C are the same as in the cases of FIGS. 5A, 5B, and 5C, respectively.

In the example of FIGS. 7A to 7C, the focus confirmation image L (Z _P ) may be formulated as the following equation (8).

また、合成部３４３は、撮影者の設定操作により「一部を重畳して並べるモード」が指定された場合に、Ｃ＝１と設定し、以下の式（９）のように図７Ａから図７Ｃのような３分割画面の中央部に重畳する画像に切り替えてもよい。

Further, the compositing unit 343 sets C = 1 when the "mode in which a part is superimposed and arranged" is specified by the setting operation of the photographer, and is shown from FIG. 7A as shown in the following equation (9). You may switch to an image superimposed on the center of the 3-split screen such as 7C.

続いて、ステップＳ４において、表示制御部３４４は、焦点確認用画像を表示装置３５に表示させる。これにより、撮影者は、カメラ２１で撮影された画像の各被写体と、合成用画像の各被写体との位置のずれの大きさ、及び方向に基づいて、所望の被写体に焦点を合わせる場合の合焦位置の調整量、及び方向を容易に認識することができる。

Subsequently, in step S4, the display control unit 344 causes the display device 35 to display the focus confirmation image. As a result, when the photographer focuses on a desired subject based on the magnitude and direction of the positional deviation between each subject of the image captured by the camera 21 and each subject of the composite image. The amount of focus adjustment and the direction can be easily recognized.

When the subject in the composite image is deviated in the negative direction of the x-axis (for example, to the left), it is in a state of being in focus on the back side of the subject (so-called rear focus), so that the image is taken. The person can recognize that the focus ring or the like should be operated to the front side (near side). Further, it can be recognized that the larger the deviation amount is, the farther away from the in-focus state.

Further, when the subject in the composite image is deviated in the positive direction of the x-axis (for example, to the right), it is in a state of being in focus toward the front side of the subject (so-called front pin). , The photographer can recognize that the focus ring or the like should be operated to the back side (far side). Also in this case, it can be recognized that the larger the deviation amount, the farther away from the in-focus state.

Further, when the subject in the composite image is not deviated, the photographer can recognize that the focus ring or the like is not required because the subject is in focus (so-called jaspin).

Subsequently, in step S5, the reception unit 345 determines whether or not the photographer has accepted the operation of adjusting the focus or zoom.

If the adjustment operation is not accepted (NO in step S5), the process proceeds to step S7. On the other hand, when the adjustment operation is accepted (YES in step S5), in step S6, the adjustment unit 346 adjusts at least one of the focus and the zoom of the camera 21 based on the adjustment operation.

Subsequently, in step S7, the determination unit 341 determines whether or not the shooting end operation has been accepted.

If the shooting end operation is not accepted (NO in step S7), the process proceeds to step S1. On the other hand, when the shooting end operation is accepted (YES in step S7), the process ends.

<Modification example 1>
Each functional unit of the control device 34 may be realized by cloud computing or the like composed of a plurality of computers.

<Effect of embodiment>
Conventional digital cameras (digital video cameras, digital still cameras) use an autofocus function that automatically focuses on the position of the subject. This autofocus function automatically focuses on the subject in the center of the shooting range or the human face recognized by image recognition.

However, with the autofocus function, the focusing position is automatically adjusted, so that the photographer cannot focus on the intended subject. Further, when the photographer wants to take an image out of focus, he / she cannot take an image out of focus intentionally. Therefore, a manual focus function that allows the photographer to manually adjust the focusing position may be used.

In this case, the photographer adjusts the focusing position by operating the focus ring or the like while visually observing the image projected on the viewfinder and visually checking the degree of blurring of the subject or the like. In this case, when a high-definition image such as 4K or 8K is taken, it may be difficult to instantly determine the in-focus position with the naked eye due to the minute blurring of the subject or the like.

In addition, in the conventional focus adjustment that relies on the amount of blurring of the outline of the subject and the contrast, the front pin that is focused in front of the targeted part and the rear pin that is focused in the back of the targeted part are used. It is difficult to distinguish. Therefore, the photographer may perform operations such as reciprocating the focus ring to make adjustments, which may make the image unsightly.

In addition, when using the autofocus function to shoot a video of a sports competition or the like that includes a plurality of subjects with a large difference in the distance in the depth direction as seen from the camera at a high-definition resolution such as 4K or 8K, the autofocus is used. The video may be out of focus and unsightly due to the processing being delayed.

In the above-described embodiment, the focus adjustment assisting device 3 is based on the first image in which the first object is photographed by the camera 21 whose focusing position is adjusted in response to the operation of the photographer, and the focusing position of the camera 21. The compositing unit 343 that synthesizes the second image whose position of the first object is shifted according to the distance to the first object and the distance from the camera 21 to the first object, and the synthesized third image are displayed on the screen. It has a display control unit 344 to be displayed. This allows the photographer to properly focus.

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments, and various modifications and variations are made within the scope of the gist of the present invention described in the claims. It can be changed. It is also possible to combine some or all of the above-mentioned examples.

1 Image capturing system 2 Imaging device 21 Camera 22 Lens 23 Adjusting device 3 Focus adjustment assisting device 31 Camera 32 Lens 33 Adjusting device 34 Control device 341 Judgment unit 342 Generation unit 343 Synthesis unit 344 Display control unit 345 Reception unit 346 Adjustment unit 35 Display apparatus

Claims (10)

A focus adjustment assisting device that assists the photographer in adjusting the focus position of the first camera.
The distance between the first image in which the first camera captures the first object and the in-focus position of the first camera to the first object, whose focusing position is adjusted in response to the operation by the photographer, is large. Indeed, the compositing unit that synthesizes the second image in which the amount of displacement of the position of the first object becomes large,
A focus adjustment assisting device including a display control unit for displaying a third image synthesized by the compositing unit on a screen. The compositing unit synthesizes the second image in which the amount of displacement of the position of the first object is increased as the distance from the first camera to the first object becomes smaller.
The focus adjustment assisting device according to claim 1. The compositing unit synthesizes the second image obtained by shifting the images taken by the second camera installed side by side with the first camera according to the focusing position of the first camera.
The focus adjustment assisting device according to claim 1 or 2. The synthesis unit generates the third image in which the first image and the second image are arranged side by side.
The focus adjustment assisting device according to any one of claims 1 to 3. Among the first region, the second region, and the third region divided from the top in the horizontal direction, the synthesis unit includes the first region and the third region of each region included in the first image. An image is set, and the third image is generated so as to set an image of a region included in the second image in the second region.
The focus adjustment assisting device according to any one of claims 1 to 4. The compositing unit generates the third image in which the first image and the second image are superimposed.
The focus adjustment assisting device according to any one of claims 1 to 5. Among the first region, the second region, and the third region divided from the top in the horizontal direction, the synthesis unit includes the first region and the third region of each region included in the first image. An image is set, and the third image is generated so that the image of the region included in the first image and the image of the region included in the second image are superimposed on the second region.
The focus adjustment assisting device according to claim 6. The compositing unit superimposes a region of the first object included in the second image with a transmittance corresponding to the clarity of the outline of the first object.
The focus adjustment assisting device according to claim 6 or 7. The synthesizing unit synthesizes an image based on the difference between the value of each pixel of the first image and the value of each pixel of the second image into the first image.
The focus adjustment assisting device according to any one of claims 1 to 8. A program for operating a computer as a focus adjustment assisting device according to any one of claims 1 to 9.

Images