JP5477777B2 - Image acquisition device - Google Patents

Description

  The present invention relates to an image acquisition apparatus for collectively displaying images taken by a plurality of cameras as one image.

  Conventionally, as an image acquisition apparatus that composes one image by combining images captured using a plurality of cameras, a virtual projection curved surface having the same size as the projection curved surface of a projector as a display device is used. And a device that combines the images by determining the shooting direction and the angle of view of a plurality of cameras so as to fill this virtual projection curved surface (see Patent Document 1) Assuming a screen, an apparatus that determines the shooting direction and angle of view of a plurality of cameras so as to fill this virtual screen and have overlapping shooting areas, and synthesizes images (see Patent Document 2) ) Etc. are known.

  However, each of the conventional image acquisition devices described above is merely a composition of a plurality of captured images, and other cameras are automatically directed in the direction of the line of sight of a person photographed by one camera. However, it does not provide a highly functional image acquisition device that captures the attention point of the person.

JP 2005-277825 A JP 2005-229369 A

  The present invention has been made in view of the above-described conventional problems, and an object of the present invention is to automatically point another camera in the direction of the line of sight of a person photographed by one camera and focus on that person. It is to realize a new function of shooting.

The invention according to claim 1 is an image acquisition device comprising a plurality of photographing means and an image processing unit for processing images photographed by the plurality of photographing means and controlling the plurality of photographing means. A first imaging unit for imaging a person, a second imaging unit for imaging a point of interest in the direction of the person's line of sight, and detecting the direction of the person's line of sight from an image of the first imaging unit. A line-of-sight direction detection unit, a point-of-interest position calculation unit that calculates the position of the point of interest from the line-of-sight direction detected by the line-of-sight direction detection unit, and an image of the point of interest position calculated by the point-of-interest position calculation unit A point-of-interest image acquisition means for acquiring from the imaging means, and a facial expression recognition means for recognizing the facial expression of the person captured by the first imaging means, wherein the facial expression recognition means When the change is captured, the line-of-sight direction Out means and performing a line-of-sight direction detection.
According to a second aspect of the present invention, in the image acquisition device according to the first aspect, the point-of-interest position calculation unit calculates a projective transformation matrix based on an elevation angle and an azimuth angle of the gaze direction detected by the gaze direction detection unit. The point-of-interest position is calculated using this method.
According to a third aspect of the present invention, in the image acquisition device according to the first or second aspect, the second photographing means is a PTZ camera capable of remotely changing an elevation angle, an azimuth angle, and a field angle in the photographing direction. The attention point position calculated by the attention point position calculating means is given by an elevation angle and an azimuth angle to be set in the PTZ camera.
According to a fourth aspect of the present invention, in the image acquisition device according to the first or second aspect, the attention point position calculated by the attention point position calculation unit is the first of the attention points captured by the second imaging unit. It is characterized by being given by coordinates in the image of the second photographing means.
According to a fifth aspect of the present invention, in the image acquisition device according to the first or second aspect, the first photographing unit is a PTZ camera capable of changing an elevation angle, an azimuth angle, and a field angle in a photographing direction. It is characterized by.
A sixth aspect of the present invention is the image acquisition device according to any one of the first to fifth aspects, further comprising sound collection means for obtaining sound in the photographing direction of the first photographing means, The gaze direction detection means detects the gaze direction when the means captures a sound or sound at a certain level or higher.
According to a seventh aspect of the present invention, in the image acquisition device according to any one of the first to fifth aspects, the line-of-sight direction detecting unit detects the line-of-sight direction when the first photographing unit captures a person. It is characterized by.
According to an eighth aspect of the present invention, there is provided an image obtained by an image acquisition device having a plurality of photographing means and an image processing unit for processing images taken by the plurality of photographing means and controlling the plurality of photographing means. An acquisition method, the step of detecting the line of sight of the person from the image of the person photographed by the first photographing means for photographing the person, and the position of the attention point of the person from the line of sight of the person A step of calculating, a step of acquiring an image of the position of the point of interest by a second photographing unit that photographs the point of interest of the person, and a method for recognizing the facial expression of the person captured by the first photographing unit A facial expression recognition step, and in the facial expression recognition step, the visual line direction detection means detects the visual line direction when a change in the facial expression of the person is captured .

  According to the present invention, a point of interest of a person can be photographed by automatically directing another camera in the direction of the line of sight of the person photographed by one camera.

It is a block diagram which shows the structure of one Embodiment of the image acquisition apparatus by this invention. It is a block diagram which shows the structure of the image processing apparatus used for one Embodiment of the image acquisition apparatus by this invention. It is a flowchart which shows the procedure of projection matrix calculation in one Embodiment of the image acquisition apparatus by this invention. It is a flowchart which shows the procedure of the attention point image imaging based on gaze direction detection in one Embodiment of the image acquisition apparatus by this invention. It is a figure for demonstrating the calculation process of a projection matrix in one Embodiment of the image acquisition apparatus by this invention. It is a figure for demonstrating the process of the attention point image imaging based on a gaze direction detection in one Embodiment of the image acquisition apparatus by this invention. It is a figure which shows the one aspect | mode of the image which synthesize | combined the audience image and the attention point image in one Embodiment of the image acquisition apparatus by this invention.

Next, an embodiment of an image acquisition device according to the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of an embodiment of an image acquisition apparatus according to the present invention.
The image acquisition apparatus includes a first photographing unit for photographing a person, for example, a fixed camera 1 in which a photographing direction is fixed in the direction of a person, and a second point for photographing a point of interest in the direction of the person's line of sight. An imaging means, for example, a PTZ (Pan-Tilt-Zoom) camera 2 capable of changing the elevation angle, azimuth angle, and angle of view of the imaging field, and an audio microphone 3 for collecting cheers and applause around the person An image processing device 4 that processes the images obtained by the fixed camera 1 and the PTZ camera 2 and controls the PTZ camera 2, and a display device 5 that displays image information obtained by the fixed camera 1 and the PTZ camera 2. And an input device 6 for the user to set the image processing device 4 and the like.

FIG. 2 is a block diagram illustrating a configuration of the image processing device 4 of the image acquisition device.
The image processing apparatus 4 includes an input / output interface 11 for communicating with the fixed camera 1 and the like, a CPU (Central Processing Unit), a ROM (Read Only Memory) in which a program is written, and a temporary storage of data. And a processing unit 12 having a RAM (Random Access Memory) or the like. In addition, the processing unit 12 detects an operation start information acquisition unit 121 for causing the processing unit 12 to start an image processing operation based on the sound captured by the audio microphone 3, and a gaze direction of a person photographed by the fixed camera 1. A gaze direction detecting unit 122 for performing the processing, a point of interest position calculating unit 123 for calculating the position of the point of interest in the gaze direction of the person, and an instruction to the PTZ camera 2 to acquire an image of the point of interest. Of interest point image acquisition means 124 and projection matrix calculation means 125 for calculating in advance the projection matrix used when the attention point calculation means 123 calculates the position of the attention point. The means is realized by the processing unit 12 including the CPU executing the program.

In the above configuration, when the sound microphone 3 captures a sound or sound having a volume level higher than a predetermined level, such as cheers or applause, the image processing operation of the processing unit 12 described below is started by the operation start information acquisition unit 121. . First, the line-of-sight direction detection unit 122 detects the line-of-sight direction of the spectator shooting with the fixed camera 1. Next, the point-of-interest position calculation means 123 calculates an elevation angle and an azimuth angle to be set in the PTZ camera 2 so as to photograph the point of interest in the line-of-sight direction, for example, an animal, with the PTZ camera 2. Finally, the attention point image acquisition unit 124 sets the elevation angle and azimuth angle of the PTZ camera 2, and the PTZ camera 2 captures the attention point.
In the above, as a method of calculating the elevation angle and azimuth angle to be set in the PTZ camera 2 from the information of the sight line direction acquired by the fixed camera 1, the angle information in the sight line direction is converted into the elevation angle and azimuth angle of the PTZ camera 2. A projection matrix can be prepared in advance by the projection matrix calculation means 125.

  As described above, since the present image acquisition apparatus is a multi-camera system using two cameras, the fixed camera 1 and the PTZ camera 2, the fixed camera 1 captures a change in the sight line direction of the audience and detects the sight line direction. A new attention point corresponding to the change can be immediately captured by the PTZ camera 2. In addition, since the image processing operation is started using the voice microphone 2 as a trigger of cheering and applause of the audience, the audience reaction and the situation of the attention point at that time are captured as images. Can be acquired at the same time. For this reason, by displaying these images on, for example, a display screen of a zoo, it is possible to guide a visitor in another location in real time about the attractions in the park.

Next, the calculation procedure of the projection matrix of the image acquisition apparatus of the present invention will be described with reference to the flowchart shown in FIG.
First, the installation location and installation direction (reference direction) of the fixed camera 1 and the PTZ camera 2 are determined (S101), and a reference point is provided within the shooting range of the PTZ camera 2 (S102).
Next, a person stands at the center of the image of the fixed camera 1, and the line of sight is directed to one reference point (S103). Subsequently, the line-of-sight direction detection unit 122 acquires the elevation angle and the azimuth angle in the line-of-sight direction from the image (S104). The elevation angle and the azimuth angle acquired at this time are angles based on the optical axis direction of the fixed camera, that is, the line-of-sight direction when a person turns his / her line of sight toward the front of the fixed camera 1.

Next, the PTZ camera 2 is directed toward the reference point, and the elevation angle and azimuth angle of the PTZ camera 2 are read from the PTZ camera 2 (S105). Then, for all reference points, it is determined whether or not the acquisition of the elevation angle and azimuth angle of the line of sight and the elevation angle and azimuth angle of the PTZ camera 2 has been completed (S106), and if not completed (S106, No), if the procedure from S103 to S105 is repeated and completed (S106, Yes), the projection matrix calculation means 125 calculates the projection matrix based on the acquired data (S107), and the process is terminated. To do.
In the description of FIG. 5 described later, a specific example of the above procedure is shown.

  In the above description, the gaze direction is detected in S104 by, for example, extracting a feature point of the face and eyes and detecting the gaze direction from the distance between the feature points (Japanese Patent Laid-Open No. 2007-265367), A method of detecting the line-of-sight direction from the position of the center of the black eye in the reference image when the five points at the center and the four corners are watched and the position of the center of the black eye in the line-of-sight detection image (Japanese Patent Laid-Open No. 2006-141862) is used. be able to. Further, in order to increase the accuracy when detecting the line of sight, a background erasing method can also be adopted. That is, it is possible to prepare a background image in the absence of a person in advance, generate a difference image between the background image and an image used for eye-gaze detection, and obtain an image of only a person without a background.

Next, an operation procedure until the image of the attention point is obtained by detecting the sight line direction of the spectator of the image acquisition device of the present invention will be described with reference to the flowchart shown in FIG. In the following description, it is assumed that the fixed camera 1 and the PTZ camera 2 have already been set by S101 described above.
First, the user installs the voice microphone 3 toward the shooting range of the fixed camera 1 (S201). Next, the operation start information acquisition unit 121 waits for the voice microphone 3 to capture a loud sound such as cheers or applause (S202). When the voice microphone 3 captures a cheer or applause (S202, Yes), the processing unit 12 acquires a photographed image of the fixed camera 1 (S203), and the line-of-sight direction detection unit 122 captures the person photographed by the fixed camera 1. Then, the elevation angle and the azimuth angle in the line-of-sight direction are detected (S204).

Next, based on the elevation angle and azimuth angle in the line-of-sight direction, the point-of-interest position calculation means 123 calculates the elevation angle and azimuth angle in the direction in which the PTZ camera 2 should be directed using the projection matrix calculated above (S205). . Finally, the attention point image acquisition unit 124 sets the elevation angle and azimuth angle of the PTZ camera 2 and acquires the image of the attention point with the PTZ camera 2 facing the attention point (S206). Thereafter, the operation start information acquisition unit 121 again waits for the voice camera 3 to capture cheers or applause (S202).
In the above-described configuration, an operation such as line-of-sight detection is performed with the voice microphone 3 capturing a cheer or applause as a trigger. However, it is detected that a person has entered the shooting range of the fixed camera 1. It is good also as a trigger, and it is good also as a trigger by detecting the motion and facial expression of the person image | photographed with the fixed camera 1. FIG.
In addition, by recording the images of the fixed camera 1 and the PTZ camera 2 and leaving the video for several seconds before and after the occurrence of the above-mentioned trigger, the recording time of the video recording is shortened, and an appropriate shutter chance is obtained from this video. Can get photos.

Next, a modification of the image acquisition device will be described.
In the first modification of the present image acquisition apparatus, both the camera used for detecting the human line of sight and the camera for photographing the attention point of the person are both PTZ cameras. In the first modified example, it is possible to detect the line of sight for a wider audience, and it is possible to expand the operation range of the image acquisition apparatus. However, in this case, it is necessary to obtain the human gaze direction by correcting the gaze direction obtained from the image of the PTZ camera with the elevation angle and azimuth angle of the PTZ camera.

  In the second modification of the image acquisition apparatus, a camera image used for line-of-sight detection is divided into several detection areas to perform line-of-sight direction detection. In this second modified example, for each detection area, a projection matrix that associates the line-of-sight direction of the person in the detection area with the direction of the camera for photographing the point of interest can be calculated. As a result, the gaze direction of various people captured by the camera used for gaze detection is captured, and for example, attention points in the gaze direction captured in the respective detection areas are sequentially captured at a certain time interval. You can shoot with the camera.

  The third modification of the present image acquisition device uses various cameras other than the PTZ camera without limiting the camera for capturing the attention point to the PTZ camera. In the third modification, the image portion of the target point in the images taken from the various cameras can be partially enlarged and displayed by the digital image enlargement method. Thereby, compared with the 1st, 2nd modification, the image of an attention point can be displayed quickly. Note that the projection matrix used in this case is obtained as a matrix indicating the correspondence relationship between the elevation angle and azimuth angle in the line-of-sight direction and the two-dimensional coordinates of the point of interest in the image of the camera for photographing the point of interest.

  The fourth modification of the present image acquisition apparatus has expression recognition means for reading an expression from an image of a person's face, and the line of sight is triggered by, for example, a change in the expression of the person's face reflected in the image to a smile. An operation such as detection is started. In the fourth modification, the operation of the image acquisition device can be started based on a finer audience response. As the facial expression recognition means used here, for example, a means (Japanese Patent Laid-Open No. 2008-9728) for extracting facial feature information by performing frequency analysis on a face image using a plurality of spatial filters can be used.

Next, the operation principle of the image acquisition apparatus will be described with reference to FIGS.
FIG. 5 is a diagram showing the relationship among the fixed camera 1, the PTZ camera 2, and the reference point when calculating the projection matrix.
As shown in FIG. 5 (1), a plurality of reference points are provided within the photographing range of the PTZ camera 2, and a person is standing in front of the fixed camera 1 so that the line of sight is directed toward one reference point. At this time, an image as shown in FIG. 5B is obtained from the fixed camera 1, and an elevation angle and an azimuth angle in the viewing direction of the audience are detected from this image. Then, the PTZ camera 2 is pointed in the direction of the reference point, and the elevation angle and azimuth angle of the PTZ camera 2 when the reference point is photographed as shown in FIG. The projection matrix is calculated by performing the above operation for all the reference points.
Here, the projection matrix to be calculated is a matrix for converting a two-dimensional vector composed of the elevation angle and the azimuth angle of the line-of-sight direction into a two-dimensional vector composed of the elevation angle and the azimuth angle of the PTZ camera 2, so that FIG. Four reference points are sufficient in S102.

FIG. 6 is a diagram showing the relationship between the fixed camera 1, the PTZ camera 2, and the attention point when the PTZ camera 2 captures the attention point in the sight line direction of the audience using the projection matrix calculated above. is there.
First, as shown in FIG. 6 (1), the audience is photographed by the fixed camera 1, the elevation angle and azimuth angle of the line-of-sight direction are detected, and the elevation angle and azimuth of the PTZ camera 2 are calculated using the projection matrix calculated as described above. Calculate and set the corner. As a result, the PTZ camera 2 faces the attention point in the sight line direction of the audience (FIG. 6 (2)), and an image of the attention point is obtained from the PTZ camera 2 (FIG. 6 (3)).
As a result of the above operation, images obtained from the fixed camera 1 and the PTZ camera 2 can be combined and displayed on the display device as shown in FIG. 7 (1) or (2), for example.

  As described above, the image acquisition apparatus of the present embodiment detects the sight line direction of a spectator with one camera, and based on the elevation angle and azimuth angle of the sight line direction, the attention point of the spectator with another camera. The elevation angle and azimuth angle necessary for photographing are calculated. For this reason, it is possible to automatically perform operations from the detection of the line-of-sight direction to the photographing of the attention point. In addition, the recording of the video recording is performed by leaving the video for several seconds before and after the occurrence of the above trigger, in order to start the detection of the line of sight and the shooting of the attention point, triggered by the cheering of the audience or applause by the voice microphone. While shortening the time, it is possible to obtain a photograph with an appropriate photo opportunity from this video.

  DESCRIPTION OF SYMBOLS 1 ... Fixed camera, 2 ... PTZ camera, 3 ... Audio | voice microphone, 4 ... Image processing apparatus, 5 ... Display apparatus, 6 ... Input device, 11 ... Input / output interface , 12, processing unit, 121, operation start information acquisition unit, 122, line-of-sight direction detection unit, 123, attention point position calculation unit, 124, attention point image acquisition unit, 125. Projection matrix calculation means.

Claims (8)

An image acquisition apparatus comprising: a plurality of photographing means; and an image processing unit for processing images photographed by the plurality of photographing means and controlling the plurality of photographing means.
A first photographing means for photographing a person;
Second imaging means for imaging a point of interest in the person's line of sight;
Gaze direction detecting means for detecting the gaze direction of the person from the image of the first imaging means;
Attention point position calculation means for calculating the position of the attention point from the gaze direction detected by the gaze direction detection means;
Attention point image acquisition means for acquiring an image of the attention point position calculated by the attention point position calculation means from the second imaging means;
Facial expression recognition means for recognizing the facial expression of the person captured by the first photographing means;
An image acquisition apparatus , wherein the gaze direction detection unit detects a gaze direction when the facial expression recognition unit captures a change in the facial expression of the person . The image acquisition device according to claim 1,
The image acquisition apparatus according to claim 1, wherein the attention point position calculation unit calculates an attention point position using a projective transformation matrix based on an elevation angle and an azimuth angle of the viewing direction detected by the viewing direction detection unit. In the image acquisition device according to claim 1 or 2,
The second photographing means is a PTZ camera that can remotely change the elevation angle, azimuth angle, and angle of view of the photographing direction,
An image acquisition apparatus characterized in that an attention point position calculated by the attention point position calculation means is given by an elevation angle and an azimuth angle to be set in the PTZ camera. In the image acquisition device according to claim 1 or 2,
2. An image acquisition apparatus according to claim 1, wherein the target point position calculated by the target point position calculating unit is given by coordinates of the target point captured by the second image capturing unit in an image of the second image capturing unit. In the image acquisition device according to claim 1 or 2,
The image capturing apparatus according to claim 1, wherein the first photographing unit is a PTZ camera capable of changing an elevation angle, an azimuth angle, and a field angle in a photographing direction. In the image acquisition device according to any one of claims 1 to 5,
Sound collecting means for acquiring sound in the photographing direction of the first photographing means;
An image acquisition apparatus, wherein the gaze direction detection means detects a gaze direction when the sound collecting means captures a sound or sound of a certain level or higher. In the image acquisition device according to any one of claims 1 to 5,
The image acquisition apparatus, wherein the gaze direction detection unit performs gaze direction detection when the first photographing unit captures a person. An image acquisition method performed by an image acquisition apparatus having a plurality of imaging units and an image processing unit for processing images captured by the plurality of imaging units and controlling the plurality of imaging units,
Detecting the direction of the line of sight of the person from the image of the person photographed by the first photographing means for photographing the person;
Calculating the position of the person's attention point from the person's line-of-sight direction;
A step of acquiring an image of the position of the attention point by a second imaging unit that images the attention point of the person ;
A facial expression recognition step for recognizing the facial expression of the person captured by the first photographing means;
An image acquisition method , wherein, in the facial expression recognition step, the visual line direction detecting means detects the visual line direction when a change in the facial expression of the person is captured .

Images