JP2001235675A - Focus control system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a focus control system by which focusing on an object area intended by an operator is easily performed or an object intended by the operator is easily followed and focused by focusing by using the distance information of an area meaning the feature amount of the object. SOLUTION: This system is constituted of a range-finding part 101, an input part 102, a feature amount extracting part 103, a corresponding position calculating part 104, a three-dimensional spatial coordinate transformation part 105, an area information generating part 106, an area information holding part 107, a controlling part 108, a table 109, a pulse generating unit 110, a lens driving part 111 and a photographing optical system 112.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a focus control system for recognizing a specific subject and detecting a focus when a moving image is taken, and more particularly to a focus control system suitable for a video camera or the like.

[0002]

2. Description of the Related Art Hitherto, a hill-climbing system, an image shift system, an infrared system or an ultrasonic system has been adopted as a focus detection system in a photographing method of a video camera or the like.

In the hill-climbing method, a sharpness of an image (subject image) is directly evaluated, a focus lens (focusing lens) is driven to change a focus, and a point where the sharpness of the image is maximized is searched for. The in-focus state of the imaging system is obtained as the in-focus position. However, in this focus detection method, it is necessary to sample a number of frames while driving the focus lens before reaching the in-focus position to obtain a sharpness evaluation value. There was a problem that it was difficult to predict.

The problem that it takes a long time for focusing is that, for example, when the above-mentioned hill-climbing type focus control system is used for a TV camera, there is a possibility that a momentary shooting opportunity may be missed in news coverage or the like.

[0005] In the focus detection of the image shift method, based on the direction and amount of shift of the image re-imaged on two line sensors arranged at positions symmetrical with respect to the optical axis of the image forming optical system,
Since the direction and amount of movement of the focusing lens required for focusing are directly calculated, focusing can be performed quickly. However, when the above-described image shift type focus control system is used in a video camera, it is necessary to split a light beam from an imaging optical system that forms an image on an imaging device and form an image at a position different from the imaging device surface. There is. For this reason, it is necessary to evaluate whether or not the image on the image sensor is in focus using an image formed at a position different from the image sensor surface. There is. In particular, a focus control system for a TV camera is required to have high accuracy (a deviation of the image plane in the optical axis direction of the photographing optical system is within several μm). It is difficult to satisfy this accuracy with an image shift type focus control system.

In the infrared or ultrasonic system, infrared or ultrasonic waves are emitted toward a subject, the reflection is input, and the intensity or time is measured to measure the distance. However, in this method, accurate distance measurement cannot be performed unless the emission of infrared rays or ultrasonic waves is properly directed to the subject.

For example, when two objects are arranged side by side, infrared or ultrasonic waves may be directed between the objects, so that the distance from the object to the object behind is measured. As a result, a hollow phenomenon in which the focus is lost may occur.

In any of the above methods, since the focus control system is based on the premise that the camera faces the subject, it is impossible to recognize and track the subject, which is different from the intention of the operator. It may happen that the target is focused.

Recently, a focus control method has been proposed which follows the gaze position information of the operator in view of the above circumstances.
The photographer does not always look at the subject at all times, and the photographer often sees information on functions displayed on the screen, such as the tape counter, battery level, and zoom position, that supplement the shooting operation. For this reason, each time the photographer's gaze position deviates from the subject, the focus state changes, and it is difficult to maintain a constant focus state for the subject intended by the photographer.

[0010]

SUMMARY OF THE INVENTION According to the present invention, an in-focus state of a photographic system with respect to a subject is detected by using distance information and feature amounts calculated from a plurality of images obtained by a plurality of cameras. It is an object of the present invention to provide a focus control system capable of performing high-speed and high-precision focus control with high accuracy, and capable of recognizing and tracking a subject as needed.

[0011]

In order to solve the above-mentioned problems, a focus control system according to the present invention employs a photographing means for photographing a subject and a principle of triangulation from a plurality of images photographed by a plurality of cameras. A distance measuring unit that calculates distance information of a shooting area using the object; a subject specifying unit that specifies a subject area from the shooting area; and control for focus control based on the distance information and the designated subject area. A focus control system comprising focus information generating means for generating information and focusing means for performing focus control of the photographing with reference to the control information is employed.

In the subject designating means, if the designating method is a method of inputting a position or area in a three-dimensional space obtained by capturing images by a plurality of cameras, the method may be such that the background is close to the subject or Even when such objects overlap, designation of the subject becomes easy.

This specification method refers to feature information obtained from a two-dimensional image and a three-dimensional image captured by a plurality of cameras, inputs a threshold value of the feature information, and specifies a subject area by the threshold value. May be used.

[0014] In addition, an arbitrary position of an image in which a photographing area is displayed is input according to a designated method, and the input position indicated by the input position is indicated by feature information obtained from a two-dimensional image and a three-dimensional image obtained by imaging with a plurality of cameras. This is also effective as a method for specifying a shooting region.

The designation method is to input a name or an identifier indicating an object to be photographed, and to recognize the object to be photographed by recognizing the object by feature information obtained from two-dimensional and three-dimensional images obtained by a plurality of cameras. However, it is also effective.

A storage means for storing characteristic information indicating the image area to be photographed together with the name or the identifier is provided, and characteristic information obtained from a two-dimensional image and a three-dimensional image obtained by photographing the photographing area with a plurality of cameras. The subject area may be specified based on the result of comparison with.

In the above-mentioned focusing information generating means, the average of the distance information indicated by the image of the designated object area, the range of the distance information indicated by the image of the object area, the distance information of the designated input position, 3 obtained by imaging with a camera
At least one selected from distance information of the center of gravity of the three-dimensional shape
It is necessary that one is included in the control information in order to focus on the subject.

In the above-mentioned focusing means, an image faithful to the photographing intention of the operator can be taken by adjusting the focus or defocus amount of the subject as required by the operator with reference to the control information.

In this focusing information generating means, the average of the distance information indicated by the image of the designated subject area, the value range of the distance information indicated by the image of the subject area, the designated input position, and the plurality of cameras At least one distance information selected from the center of gravity of a three-dimensional shape obtained by imaging is tracked for each frame, and at least one selected from a speed, an acceleration, and a predicted position of a subject after a lapse of a predetermined time is generated as control information. It is effective that the focusing means predicts the focus in advance with reference to the control information to perform the focusing so as to perform quick and accurate focus control.

[0020] At least one feature information selected from feature information obtained from a two-dimensional image and a three-dimensional image obtained by imaging a plurality of cameras of the subject area designated by the subject designating means is stored, and the subject moves. In this case, it is also effective to track the object area based on the feature information and perform the focus control to perform quick and accurate focus control.

A plurality of subject areas are designated by the subject designating means, and at least one selected from feature information obtained from two-dimensional images and three-dimensional images obtained by imaging the designated subject areas with a plurality of cameras. The feature information is stored, even when the plurality of subjects move, the plurality of subject areas are tracked by the feature information, and focus control can be performed by switching the subject area to be focused as needed by the operator. It is equally effective.

If the target object of the shooting is a sports broadcast and broadcast, and the focus control is performed by following at least one moving subject selected from a player, a referee, a ball, a puck, and equipment used in the sport, Sports broadcasting and the like can be performed more effectively.

The object of shooting is relaying and broadcasting news reports, and if a system that controls the focus of the entire person or a subject that is a specific part of the person according to the intention of the news, the center of the incident is It is easy to focus on a person or a specific part thereof, and it is also possible to perform a defocusing process on an object that is not preferable in the press.

The object of the shooting is a sequence shooting of a TV drama and a movie, and if there is a system for controlling the focus on the characters or the appearances according to the necessity of production, efficient drama shooting becomes possible. .

[0025]

With the above-described system, focus control can be performed quickly and accurately for a subject to be photographed, and for a moving subject as long as the subject is within the photographing area. Becomes

[0026]

FIG. 1 is a block diagram showing an embodiment. Distance measuring section 101, input section 102, feature quantity extracting section 103, corresponding position calculating section 104, three-dimensional spatial coordinate converting section 105, area information generating section 106, area information holding section 10
7, control unit 108, table 109, pulse generator 11
0, a lens driving unit 111, and a photographing optical system 112.

The distance measuring unit 101 generates two-dimensional image data and three-dimensional image data (distance image data) by taking an image of a photographing area and performing stereo processing. 2 and 3
As shown in (1), a distance measuring system imaging unit 201 and a stereo processing unit 202 are provided. Here, the distance measuring system imaging means 2
Reference numeral 01 denotes a structure in which nine cameras including the camera 301 are fixed to the base plate 302 at a predetermined base distance as shown in FIG. Also, stereo processing means 2
02 calculates the distance information of the image (pixel) in the photographing area by a predetermined algorithm using the principle of triangulation by calculating the difference between the distance between the bases and the image (two-dimensional image data) obtained by the plurality of cameras. An arithmetic circuit that generates distance image data, that is, three-dimensional image data.

In this embodiment, nine cameras are employed. However, the number of cameras is not limited to nine as long as a plurality of images of the photographing area can be obtained with two or more cameras. Although the arithmetic circuit is used for the stereo processing, the same processing may be performed by a software program.

The input unit 102 comprises a position input means 401 and an intention input means 402 as shown in FIG. 4, and the operator inputs the position of a target to be a subject and a focus intention for the subject. As the position input means 401, a touch panel laminated on a liquid crystal display was employed. When inputting a position on the screen, an image of the shooting area is displayed on a liquid crystal display, and an operator touches an arbitrary position on the stacked touch panel with a finger or the like. The touch position is output to the corresponding position calculation means 104.

In the case of inputting the position of the space, a three-dimensional coordinate space diagram showing the photographing area is displayed on the liquid crystal display, and the operator touches an arbitrary position on the stacked touch panel with a finger or the like. The touch position is output to the three-dimensional space coordinate conversion unit 105.

The intention input means 402 includes a focus / defocus instruction input means 405 and a focusing priority input means 4
06, an identifier input means 407. The focus / defocus instruction input means 405 is an instruction input for determining whether to focus on the specified subject or to shift the focus from the in-focus state by a predetermined amount for blurring, temporarily displaying a menu screen on the liquid crystal display, Input by touch operation of. The focus priority input unit 406 similarly inputs the focus priority from the menu screen when a plurality of subjects are specified. The identifier input means 407 assigns an identifier to a feature amount indicating a specific target in advance and registers the characteristic amount, and similarly inputs the identifier from the menu screen as a focus target. For example, if a person is to be taken as a subject, a "person" is used for the feature amount (skin color, three-dimensional head shape, etc.) indicating the person.
If you want to specify a person as the object to be focused, specify and input "person" from the menu screen. Although the touch panel is used here, the position input means of the present invention is not limited to this, and may be a mouse operation on a monitor or an input using a keyboard.

As shown in FIG. 5, the feature amount extracting unit 103 has a configuration of a feature extracting unit 501, a recognizing unit 502, an extraction information holding unit 503, and a shape parameter designating unit 504.
The feature amount necessary for generating the imaging region information is continuously extracted from the imaging area. The feature extracting unit 501 receives the two-dimensional image data output from the distance-measuring system imaging unit 201 and the three-dimensional image data output from the stereo processing unit 202, and extracts a feature amount according to a predetermined algorithm. Here, a three-dimensional shape, a three-dimensional texture, and a skin color are extracted as feature amounts. The recognizing unit 502 refers to the above-described threshold value regarding the feature amount designated by the shape parameter designating unit 504 and the feature information held in the extraction information holding unit 503 to recognize a region that is a candidate for a subject region, and Each time, the position, speed, and characteristics of the area are detected,
The information is transmitted to the area information generation unit 106.

The feature quantity extraction unit 103 is realized by executing a predetermined program stored in a storage device by a central processing unit. Of course, the present invention may be realized by an arithmetic circuit that performs the same data processing as the above algorithm or program, and the present invention is not limited to these.

The corresponding position calculating section 104 comprises a two-dimensional corresponding position calculating means 601 and a three-dimensional corresponding position calculating means 602 as shown in FIG. The two-dimensional corresponding position calculating means 601 calculates two-dimensional coordinate data of the photographing area from the touch position on the screen transmitted from the position inputting means and the two-dimensional image transmitted from the distance-measuring imaging means, Send to generation unit 106. Three-dimensional corresponding position calculating means 602
Calculates the three-dimensional coordinate data of the photographing area from the touch position on the screen transmitted from the position input means and the distance image information transmitted from the stereo processing means, and sends the data to the area information generating unit 106. The corresponding position calculation unit 104 is realized by executing a predetermined program stored in the storage device by the central processing unit. Of course, the present invention may be realized by an arithmetic circuit that performs the same data processing as the above algorithm or program, and the present invention is not limited to these.

The three-dimensional space coordinate conversion unit 105 converts the touch position input by the space position input unit 404 into three-dimensional coordinate data, and sends it to the area information generation unit 106. The three-dimensional space coordinate conversion unit 105 is realized by a conversion table between a touch position and three-dimensional coordinates. Of course, it may be realized by an arithmetic circuit that performs the same data processing, and the present invention is not limited to these.

When the above touch panel is touched, that is, when an input is made in the input unit 102 such as at the start of photographing, the initial coordinates specified by the corresponding position calculation unit or the three-dimensional space coordinate conversion unit are used. Is received from the area information generating unit 106, and information on the area indicating the feature amount indicated by the initial coordinates is held in the extracted information holding unit 504 as initial feature information. After that, the extraction information holding unit holds the information of the area recognized for each frame in time series, so that the recognition unit 502 continues to recognize the subject at the touch position even when the subject is moving. Can be.

The algorithm for extracting these feature amounts is known, and can be satisfactorily performed by conventional techniques. Although the three-dimensional shape, the three-dimensional texture, and the flesh color are employed as the feature values, the feature value used in the present invention is not limited to this, and may be a contour line, an edge, or a luminance.

In the area information generating section 106, the recognizing means 50
2 specifies an object area by an algorithm of background separation or individual identification based on a position occupied by a candidate area of an object area that is recognized and transmitted for each frame, and a characteristic, and the speed and the speed transmitted from the recognition means. The object area information is generated together with the focus / defocus instruction from the intention input means, the priority of focusing, and the identifier. The specified subject area is stored in the area information holding unit 107, and is referred to by the focus control unit 108 as needed.

The area information holding unit 107 is a stack type storage device, and the types of data held are the position coordinates, speed, feature amount, and control code of the object area. Here, the control code is a focus / defocus instruction, an in-focus priority, and an identifier designated by the intention input means.

The control unit 108 refers to the distance image information sent from the distance measuring unit 101 and the area information held in the area information holding unit 107 to predict the current distance information of the object area and predict the moving subject. The distance information to be calculated is calculated. The prediction of the distance information is effective when performing focus control on a rapidly moving subject in sports broadcasting. In addition, it is desirable to use, as the distance information, an average value of the distances indicated by the pixels of the object area or the distance of the center of gravity of the object area.

The table 109 replaces the distance information received from the control unit with drive pulse information. The pulse generator 110 generates a pulse signal according to the drive pulse information. The lens driving unit 111 drives the lens by an amount corresponding to the pulse signal. The photographing optical system focuses on the subject.

If the focus control system of the present system is used,
When it is desired to focus on the ball in a live broadcast of soccer, the ball is first touched on the liquid crystal screen, which is the position input means. Next, open the menu screen as the intention input means,
Select and enter the highest priority for focusing. With such a setting operation, the photographer can take a video focused on the ball simply by adjusting the direction of the camera so that the ball enters the shooting area.

Alternatively, if the distance measuring means composed of nine cameras of the present invention can obtain the distance information of each pixel finely, the ball may be recognized by the unevenness (three-dimensional texture) of the skin making the ball. . In this case, the feature amount of the unevenness of the skin forming the ball is registered in advance as an identifier “ball”, and “ball” may be selected and input by the identifier input of the menu screen as the intention input means of the input unit. When the soccer relay starts, if the operator of the camera maintains the shooting area in the area where the ball is, the focus control is automatically performed based on the distance information of the image area indicating the characteristic amount of the ball.

If the person who is the center of the incident in the news report is a minor, the position input means designates the face of this person, and the focus / menu of the menu screen which is the intention input means of the input unit is designated.
If you enter defocus in the defocus instruction input, you can stay in the shooting area firmly even in a confused situation where the whirlpool person is messed up, but blur without focusing on that person's face The intention of giving consideration to the press.

In the shooting of a drama or the like, a sequence in which a plurality of characters move while intersecting traffic lines frequently occurs. Focus on character according to the intention of the production
To character B. In a scene where A and B are running in different directions with their respective movement lines, skill is required for focusing.
If A and character B are respectively registered in the system of the present invention together with their feature amounts by name (identifier), it is possible to handle only by calling the menu screen of the identifier input means and switching the identifier to be focused. there is a possibility.

[0046]

Since distance information for each pixel can be obtained from a plurality of images taken by a plurality of cameras, it is possible to easily focus on a subject area intended by the operator. Further, since the distance information can be obtained for all the pixels in the shooting area, the subject intended by the operator can be easily tracked and focused by focusing with the distance information of the area indicating the feature amount of the subject.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment.

FIG. 2 shows details of a distance measuring unit.

FIG. 3 is a hardware configuration of a distance measuring unit.

FIG. 4 is a configuration of an input unit.

FIG. 5 is a configuration of a feature amount extraction unit.

FIG. 6 is an area information generation unit.

[Explanation of symbols]

 Reference Signs List 101 distance measurement unit 102 input unit 103 feature amount extraction unit 106 area information generation unit 108 control unit 112 photographing optical system

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2H051 BA17 BB01 BB07 CE23 DA03 DA04 DA05 DA07 DA08 DA10 DA12 DA13 DA15 DA16 DA18 DA19 DA20 DA21 DA23 DA25 DA28 DA30 DA31 DA37 DA38 GA12 GA15 GA16 GB20 2H059 AA10 AA12 5C022 AB24 AB61 AB68 5C022 AA20 AB04

Claims (14)

[Claims] 1. A photographing means for photographing a subject, a distance measuring means for calculating distance information of a photographing area using a principle of triangulation from a plurality of images taken by a plurality of cameras, Subject designating means for designating a subject area from; focus information producing means for producing control information for focus control from the distance information and the designated subject area; and performing the photographing with reference to the control information. A focus control system comprising focusing means for performing a focus control of (1). 2. A method according to claim 1, wherein said subject designation means is a method of inputting a position or area in a three-dimensional space obtained by imaging with a plurality of cameras. Focus control system. 3. A method according to claim 1, wherein said subject designating means refers to feature information obtained from a two-dimensional image and a three-dimensional image obtained by imaging with a plurality of cameras, and inputs a threshold value of said feature information. 2. The focus control system according to claim 1, wherein a subject area is designated by a threshold value. 4. In the subject designating means, a designation method is to input an arbitrary position of an image in which a photographing area is displayed, and to obtain a two-dimensional image and a three-dimensional image obtained by imaging with a plurality of cameras.
2. The focus control system according to claim 1, wherein the focus control system is configured to specify a subject area indicated by the input position based on feature information obtained from a two-dimensional image. 5. In the subject designating means, the designation method is to input a name or an identifier indicating a subject to be photographed, and to specify the subject to be photographed based on feature information obtained from two-dimensional images and three-dimensional images obtained by imaging with a plurality of cameras. 2. The focus control system according to claim 1, wherein the method is a method of identifying a subject area by recognizing a focus. 6. A storage means for storing characteristic information indicating the image area of the photographing target together with the name or identifier, wherein the photographing area is obtained from a two-dimensional image and a three-dimensional image obtained by photographing the photographing area by a plurality of cameras. 6. The focus control system according to claim 5, wherein an object area is specified based on a result of comparison with the characteristic information. 7. The focus information generating means, wherein the average of the distance information indicated by the image of the designated object area, the range of the distance information indicated by the image of the object area, the distance information of the designated input position , Obtained by imaging with a plurality of cameras
At least one selected from distance information of the center of gravity of the three-dimensional shape
2. The focus control system according to claim 1, wherein one of the two is included in the control information. 8. The focus according to claim 1, wherein said focusing means adjusts a focus or defocus amount for a subject as required by an operator with reference to said control information. Control system. 9. The focusing information generating means, wherein: an average of the distance information indicated by the image of the designated subject area; a range of the distance information indicated by the image of the subject area; a designated input position; At least one distance information selected from the center of gravity of a three-dimensional shape obtained by imaging with a camera is tracked for each frame, and at least one selected from a speed, an acceleration, and a predicted position of a subject after a lapse of a predetermined time is controlled. 2. The focus control system according to claim 1, wherein the focusing unit predicts a focus in advance with reference to the control information to perform focusing. And storing at least one piece of feature information selected from feature information obtained from a two-dimensional image and a three-dimensional image obtained by capturing images of a subject area designated by the subject designating means with a plurality of cameras. 2. The focus control system according to claim 1, wherein even when the object moves, the object area is tracked by the feature information and focus control is performed. 11. A method according to claim 1, wherein said subject designating means designates a plurality of subject areas, and at least selected from feature information obtained from a two-dimensional image and a three-dimensional image obtained by imaging the designated subject area with a plurality of cameras. One piece of feature information is stored, even if the plurality of subjects move, the plurality of subject areas are tracked by the feature information, and focus control is performed by switching the subject area to be focused as required by the operator. 2. A focus control system according to claim 1, wherein 12. The object of photographing is sports broadcasting and broadcasting, and focus control is performed by following at least one moving subject selected from a player, a referee, a ball, a puck, and equipment used in the sport. The focus control system according to claim 1, 9 or 10, wherein: 13. The object of photography is relay and broadcast of news reports, and focus control is performed on a subject who is a whole person or a specific part of the person according to the intention of the news report. Claims 1 and 8,
Item 10. The focus control system according to Item 11, wherein 14. The method according to claim 1, wherein the object of photographing is a sequence photographing of a television drama and a movie, and focus control is performed on a character or an object as required for the production. Term, term 8, term 10,
Item 12. The focus control system according to Item 11.