JP3129595B2 - Image processing device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for specifying a desired target object from image data, for example, an image processing apparatus used for a videophone, a video conference apparatus, and the like.

[0002]

2. Description of the Related Art Conventionally, in a videophone or video conference system, a central object is enlarged by a method such as enlarging a central object in an image captured by an imaging device such as a CCD (charge coupled device) camera. Detects the center object using the method of projecting the object preferentially or the movement or sound information of the object in the captured image data, improves the resolution of the center object, and transmits a clear image A way to do that has been developed. For example, IEICE Journal, 1
In the conventional image processing apparatus described in December 1999, p. 1356, "Developing a color moving picture coding apparatus",
By extracting information on the presence or absence of movement from the image displayed on the videophone, and making the moving part a region of the person image and compressing and transferring that person image with priority over the background image, it can be displayed on the caller's display monitor. A method of clearly displaying a person image has been disclosed.

[0003] Other conventional image processing apparatuses include:
In IEICE IE92-49, "Control of Moving Image Coding by Sound Source Position Information in TV Conference", a plurality of microphones are installed on a conference seat with a plurality of persons, and audio signals captured by each of the microphones are set. There is disclosed a method of estimating a speaker's position from a plurality of persons by estimating a speaker's position based on a correlation between positions between microphones and a sound signal level.

[0004]

In the above-described image processing apparatus for preferentially transferring a person image, a desired person is simply determined based on the presence or absence of motion. Could not be identified. In other words, when another person moves behind the target person, when there is a moving person other than the person in the image, or when a plurality of people exist in the image, the center Is a moving object other than the target person (for example, a speaker in a plurality of persons), and the target person could not be preferentially compressed and transferred.

Further, in an image processing apparatus for specifying a speaker using voice information, only a speaker can be specified, but a desired target object cannot be specified. In other words, in a videophone call or a videoconference, the target object (the image most needed by the caller) is not only the face image of the speaker, but also some text, drawing, chart, or the like. In some cases, a display frame such as a blackboard is a target object whose center is a target object, and in some cases, an object (a product or a model diagram of a topic, etc.) is a target object. In this way, when the target object is a person other than the speaker, tracking the speaker is rather a disadvantage. Furthermore, since voice information is used, a plurality of microphones are necessarily required, and further, it is necessary to maintain a positional relationship between the microphones and an image input device such as a CCD camera. Was.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and has as its object to provide an image processing apparatus capable of specifying a desired target object.

Another object of the present invention is to provide an image processing apparatus capable of specifying the position of a speaker.

Still another object of the present invention is to provide an image processing apparatus capable of measuring the position of a stationary object.

Yet another object of the present invention is to provide an image processing apparatus capable of specifying the position of an elongated moving object.

[0010] Still another object of the present invention is to provide an image processing apparatus capable of specifying each person even when a plurality of persons are photographed in an overlapping manner.

Still another object of the present invention is to provide an image processing apparatus capable of specifying the position of an object even when the pointing rod or the like is not used for the object.

Still another object of the present invention is to provide an image processing apparatus capable of specifying the position of an object even when the object is pointed from all directions by a pointing rod or the like.

[0013]

In the information processing apparatus according to the present invention, a predetermined object is analyzed by analyzing image data.
An image processing apparatus for detecting , comprising: a difference means for obtaining time difference data of continuous image data including a plurality of moving objects; an addition means for cumulatively adding the time difference data; and a time difference data cumulatively added by the addition means. Is calculated based on the accumulated time difference data for each of a plurality of moving objects.
Variable binarizing means for binarizing with a threshold value for each of the plurality of moving objects, and detecting means for detecting the positions of the plurality of moving objects based on the data binarized by the binarizing means. Including.

[0014] The image processing apparatus according to the second aspect is the first aspect.
In addition to the configuration of the invention described in, the image data represents a person.
Includes to data, the two-value of image data, among the data and face representing the contour of the face of the person by binarizing means
Based on at least one of the data representing the outline of the element
A first specifying means for specifying a first area including a person's face, a second specifying means for specifying a second area including a lip of a person from the first area, and a movement amount of a lip in the second area detecting a third specifying means and further including <br/> No for detecting the position of the speaker.

According to a third aspect of the present invention, there is provided the image processing apparatus according to the first aspect.
In addition to the configuration of the invention described in the above, the image data represents a stationary object.
Includes data representing still a differentiating means for generating differential image data by differentiating the image data, based on the projection means, the extreme value of the projection data to create projection data by projecting the differentiated image data in a predetermined direction further comprising a stationary object detection means for detecting the position of the object.

An image processing apparatus according to a fourth aspect of the present invention provides an image processing apparatus according to the first aspect.
In addition to the configuration of the invention described in, the image data is elongated
Including data representing the body, a motion area detecting means for detecting a motion area including an elongated moving object from the image data, and differentiating means for differentiating the image data to create differential image data,
Area extracted based on differential image data from motion area
A projection means for the density data generating projection data projected to the predetermined direction, further comprising a moving object detecting means for detecting a position of the elongated moving objects based on the extreme value of the projection data.

An image processing apparatus according to a fifth aspect of the present invention provides an image processing apparatus according to the first aspect.
In addition to the configuration of the invention described in (1), a motion area detecting means for detecting a motion area in an image composed of continuous image data using time difference data of luminance data in the continuous image data, Binarization means for binarizing luminance data with a predetermined threshold value and outputting binarized data; hue value calculating means for calculating hue values based on luminance data and color data of image data; And a predetermined constant value to calculate a hue area having a predetermined hue, and an object for calculating a position of a predetermined object in the hue area based on the hue area and the binarized data further comprising a goods position calculating means.

An image processing apparatus according to a sixth aspect of the present invention is the first aspect of the invention.
In addition to the configuration of the present invention, a motion area detecting means for detecting a motion area in an image composed of continuous image data by using time difference data of the continuous image data, A binarizing means for binarizing the binary data according to the threshold value and outputting the binarized data; performing a contraction and expansion processing on the binarized data; A narrow area calculating means for calculating a narrow area of the binarized data based on a difference from the binarized data; obtaining neighboring area connection information for the narrow area; further comprising a elongated area calculating means for calculating an elongated region having an area equal to or larger than the value.

An image processing apparatus according to a seventh aspect is the first aspect.
In addition to the configuration of the invention described above, differentiating means for differentiating image data and calculating differential image data in the horizontal and vertical directions, and differentiating in the horizontal and vertical directions with respect to an area near the center of the image composed of the image data Horizontal and vertical projection means for projecting image data on a vertical axis and a horizontal axis, respectively, and outputting horizontal and vertical projection data; and using a horizontal and vertical projection data to determine whether or not there is a predetermined object near the center of the image Determining means for determining the
The image corresponding to the predetermined object determined to be near the center
Further comprising an extraction means for extracting image data.

[0020] The image processing apparatus according to the eighth aspect is the first aspect.
In addition to the structure of the invention according to, from the color data of the image data and the hue value calculating means for calculating hue values of the horizontal and vertical directions with respect to the region near the center of the composed image from the image data in the horizontal and vertical directions Horizontal and vertical hue projection means for projecting hue values on the horizontal axis and the vertical axis, respectively, and outputting horizontal and vertical hue projection data, and a predetermined object near the center of the image using the horizontal and vertical hue projection data. > A judgment means for judging whether there is an object and a judgment means
Corresponds to the target object determined to be near the center of the image
Further comprising an extraction means for extracting the image data to be.

[0021]

According to the first aspect of the present invention, the positions of the plurality of moving objects included in the image data are detected by binarizing the time difference data with a predetermined threshold value for each of the plurality of moving objects. can do.

In the image processing apparatus according to the second aspect,
The movement amount of the lips of a person is detected from the image data, and the position of the speaker can be detected.

In the image processing apparatus according to the third aspect,
By differentiating the image data and projecting the image data in a predetermined direction, the edge of the stationary object can be specified, and the position of the stationary object can be detected.

In the image processing apparatus according to the fourth aspect,
The position of the elongated moving object can be detected by calculating the extreme value of the projection data in the motion area including the elongated moving object.

In the image processing apparatus according to the fifth aspect,
A motion area in an image is detected using the time difference data of the luminance data, and the luminance data corresponding to the motion area is binarized by a predetermined threshold value. Can be specified. Further, a hue region having a predetermined hue value is detected,
A portion having a predetermined color in the motion area can be specified.

In the image processing apparatus according to the sixth aspect,
Binary data representing a plurality of motion regions is detected by detecting a motion region in the image using time difference data of the continuous image data and binarizing luminance data corresponding to the motion region with a predetermined threshold value. Can be obtained. The binarized data is subjected to erosion and dilation processing to calculate a narrow region of the binarized data, obtain neighboring region connection information for the calculated narrow region, and use this neighboring region connection information. It is possible to specify the position of an elongated region having a small width and an area equal to or larger than a predetermined value.

[0027] In the image processing apparatus according to the seventh aspect,
The differential image data is projected in the horizontal and vertical directions, edge components in the image are detected based on the horizontal and vertical projection data, and it is determined whether or not a predetermined object is present near the center of the image based on the edge components. can do.

[0028] In the image processing apparatus according to the eighth aspect,
The hue value is projected in the horizontal and vertical directions, and it is possible to determine whether or not there is a predetermined object near the center of the image based on the difference in the hue amount.

[0029]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image processing apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the first embodiment of the present invention.

In FIG. 1, the image processing apparatus includes an A / D converter 1, an image memory 2, an absolute difference section 3, a comparison section 4, a threshold value determination section 5, an X-axis density projection section 6, a continuous area peak detection. Sections 7, 10, continuous area independent comparison sections 8, 11, time direction accumulation section 9, object area width extraction section 12, calculation object selection section 13,
Y-axis density projection unit 14, maximum Y-axis projection coordinate extraction unit 15, face width extraction unit 16, face feature extraction determination unit 17, speaker determination unit 1
8, an extraction coordinate determination unit 19, an edge detection unit 20, a display frame detection unit 21, a pointing rod detection unit 22, and an extraction target object priority designation unit 23.

Hereinafter, the operation of the image processing apparatus will be described. First, a video signal photographed by an imaging device such as a CCD camera is input to an A / D converter 1 and converted from an analog signal to a digital signal. The image memory 2, the absolute difference unit 3, the threshold value determination unit 5, It is output to the edge detection unit 20.

The image memory 2 stores the video signal converted to a digital signal, and outputs the video signal converted to a digital signal to the absolute difference section 3 after elapse of k frame time. Here, k is a value that changes according to the situation. For example,
When the motion amount of the moving object included in the video signal is large, a small value is used, and when it is small, a large value is used.

The absolute difference unit 3 performs an absolute difference between the image data output from the A / D converter 1 and the image data delayed by k frame times by the image memory 2, and sends the time difference data to the comparison unit 4. Output.

On the other hand, threshold value determining section 5 determines a predetermined threshold value according to the image data output from A / D converter 1 and outputs the threshold value data to comparing section 4.

The comparing section 4 compares the time difference data output from the absolute difference section 3 with the threshold data output from the threshold value determining section 5 to create a binary difference signal, Projection unit 6, Y-axis density projection unit 14, face width extraction unit 16,
Output to the display frame detecting section 21 and the pointing rod detecting section 22.

The X-axis density projection unit 6 performs density projection (cumulative addition of the same X coordinate value) on the X-axis (horizontal direction of the photographed image) on the difference binary signal output from the comparison unit 4. ) Is obtained. FIG. 2 is a diagram for explaining processing from density projection processing to object area width extraction processing. When a plurality of moving objects are included in the image as shown in FIG. 2A, the density projection data appears as a plurality of continuous regions as shown in FIG. 2B. In the plurality of continuous regions, the maximum value of the density projection data in each continuous region changes according to the size of the moving object on the image. Therefore, as described later, by binarizing the density projection data according to each local maximum value, it becomes possible to detect the positions of a plurality of moving objects regardless of the size of the moving object. The density projection data created by the X-axis density projection unit 6 is output to the continuous region peak detection unit 7 and the continuous region independent comparison unit 8.

The continuous area peak detector 7 determines each local maximum value of the density projection data output from the X-axis density projector 6, for example, peak1 and peak2 in the case shown in FIG.
Is multiplied by 1 / n (n is a predetermined number) and is output to the continuous area independent comparing section 8.

The continuous area independent comparing section 8 calculates a value obtained by multiplying a value obtained by multiplying 1 / n of each maximum value output from the continuous area peak detecting section 8 by X
The density projection data output from the axial density comparison unit 6 is compared with the density projection data to binarize the density projection data. The binarized density projection data is as shown in FIG. In the above processing, by binarizing a plurality of continuous regions with a threshold value corresponding to each local maximum value, the positions of the plurality of moving objects can be determined regardless of the magnitude of the amount of movement of different moving objects. It becomes possible to detect.

The binarized density projection data is input to the time direction accumulating section 9 and is cumulatively added for a predetermined time as shown in FIG. The above-described process of cumulative addition has an effect of extending the detection time for the case where the amount of motion per unit time of the moving object is small, and it is possible to more appropriately detect the moving region even with the moving object having a slow moving amount. Become. The density projection data cumulatively added by the time direction accumulator 9 is output to the continuous area peak detector 10 and the continuous area independent comparator 11.

The continuous area peak detector 10 calculates the maximum value of the cumulatively added density projection data input from the time direction accumulator 9, for example, peak in the case shown in FIG.
3, peak4 is obtained for each continuous area. Each obtained maximum value is output to the continuous area independent comparison section 11 after multiplying by 1 / m (m is a predetermined value).

The continuous area independent comparing section 11 compares the value obtained by multiplying each local maximum value output from the continuous area peak detecting section 11 by 1 / m with the density projection data output from the time direction accumulating section 9, and The density projection data is binarized as shown in FIG. In other words, by binarizing a plurality of continuous areas of the density projection data with a threshold value corresponding to each local maximum value, the positions of the plurality of moving objects can be determined regardless of the magnitude of the amount of movement of the different moving objects. It becomes possible to detect.
The binarized density projection data is output to the object area width extraction unit 12.

As shown in FIG. 2E, the object region width extracting unit 12 sets the X coordinate (Xa1, Xb
1) and (Xa2, Xb2) are output to the calculation object selection unit 13. With the above processing, it is possible to accurately detect the positions of a plurality of moving objects regardless of the magnitude of the amount of movement of the moving objects.

The calculation object selection unit 13 uses the X coordinate value for each continuous region output from the object region width extraction unit 12,
A desired moving object is selected from a plurality of moving objects, and the X coordinate values (Xa, Xb) of the moving object are output to the Y-axis density projection unit 14 and the face width extracting unit 16.

In the above processing, the number of moving objects is described as two, but the same processing can be performed even when the number of moving objects is N (N> 2).

Next, a process for determining the speaker will be described. FIG. 3 is a diagram for explaining the speaker determination process.

First, the Y-axis density projection unit 14 sets the X coordinate (X
a, Xb) is subjected to Y-axis (vertical direction of image) density projection of the binary difference signal output from the comparison unit 4 for the inclusion section area (X coordinate value = Xa to Xb). Is output to the maximum Y-axis projected coordinate extracting unit 15.

As shown in FIG. 3, the maximum Y-axis projected coordinate extracting unit 15 is a top position Ytop of the Y coordinate of the maximum vertical length in the density projection data output from the Y-axis density projected unit 14.
And outputs it to the face width extraction unit 16, the face feature extraction determination unit 17, and the speaker determination unit 18. Here, Ytop is the top coordinate of the person.

The edge detecting section 20 differentiates the video signal converted into a digital signal by the A / D converter 1 in the vertical and horizontal directions of the image, and separates the vertical differential image data and the horizontal differential image data. Find, face width extraction unit 1
6. Output to the facial feature extraction determination unit 17 and the speaker determination unit 18.

The face width extraction unit 16 extracts the binary difference signal output from the comparison unit 4, the X coordinate (Xa, Xb) of the moving object output from the calculation object selection unit 13, and the maximum Y axis projection coordinate. The top coordinate Ytop of the person output from the unit 15 and the vertical differential image data output from the edge detection unit 20 are input. The face width extraction unit 16 uses the top coordinate Ytop of the person as a search start point and sets the X coordinate (X
a, Xb), the difference binarized signal is searched downward in the inclusive section area, and the X-coordinates (Lcor, Rco, Rco, Rco) of the face width of the person as shown in FIG.
r), the facial feature extraction determination unit 17 and the speaker determination unit 1
8, output to the display frame detecting section 21 and the pointing rod detecting section 22.

The face feature extraction determination unit 17 outputs the person's top coordinate Yto output from the maximum Y-axis projection coordinate extraction unit 15.
p, the X-coordinate (Lcor, Rcor) of the face width of the person output from the face width extraction unit 16 and the horizontal and vertical differential image data output from the edge detection unit 20 are input. The face feature extraction determination unit 17 determines the X coordinate (Lco
r, Rcor) and the top coordinate Ytop, determine the candidate head position of the person, examine the horizontal and vertical differential image data at that position, determine whether or not the person is the head, and state the determination result. To the person determination unit 18. The face feature extraction determination unit 17 analyzes whether or not information corresponding to a vertical line on the cheek and a horizontal line such as eyebrows and eyes, which is one of the facial feature amounts of the person, is included, and the face characteristic extraction determination unit 17 determines whether the person's head It is possible to accurately determine whether or not there is.

Next, a description will be given of a method of specifying a speaker from the plurality of obtained persons. FIG. 4 is a block diagram illustrating a configuration of the speaker determination unit illustrated in FIG.

In FIG. 4, the speaker determination section includes a lip area calculation section 31, a horizontal differential Y-axis density projection section 32, a projection width accumulation value calculation section 33, and a lip change amount determination section 34.

When the facial feature extraction determination unit 17 determines that the moving object includes the head of a person, the following processing is performed. First, the lip area calculation unit 31 calculates the vertical position (MY1, MY2) where the lip shown in FIG. 3 exists using the X coordinate (Lcor, Rcor) and the top coordinate Ytop of the input human head as follows. It is determined by the formula.

Xlen = Rcor-Lcor mh = Xlen.MH / XL MY1 = (Ytop + Xlen) -mh / 2 MY2 = (Ytop + Xlen) + mh / 2 where XL: head reference width, MH: lip reference height , The horizontal differential Y axis density projection unit 32 calculates the X coordinate (Lcor, Rcor) of the head and the vertical position (MY1, M
Y2), the horizontal differential image data Xed detected by the edge detection unit 20 for the region (the lip presence region)
ge is density-projected on the Y-axis, cumulatively added, and the cumulatively added data is output to the projected width cumulative value calculation unit 33.

The projection width cumulative value calculation unit 33 calculates the vertical width Mh of the accumulated horizontal differential image data and the area value Sm of the cumulative addition area, and outputs them to the lip change amount determination unit 34.

The lip change amount determination unit 34 detects whether or not a person is speaking (whether or not the lip is moved up and down) by detecting the change amount of the input vertical width Mh and area value Sm with time. Is determined, and the determination result is output to the extraction coordinate determination unit 19. In this process, by examining how much the horizontal differential value of the lip existence region has changed, it is possible to indirectly know how much the lip, which contains a large number of horizontal differential values, moves up and down. Therefore,
Among a plurality of persons to be obtained, a person whose change amount is equal to or more than a predetermined value and whose change time is the most often for a predetermined time or more is determined as a speaker.

According to the above-described processing, it is possible to determine the speaker from a plurality of persons using only the video signal which is image data.

Next, the operation of the display frame detecting section 21 will be described in detail. FIG. 5 is a block diagram showing a configuration of the display frame detection unit shown in FIG. FIG. 6 is a diagram for explaining the operation of the display frame detection unit shown in FIG.

In FIG. 5, the display frame detecting section includes a horizontal / vertical differential value density projecting section 51 and a projected maximum value coordinate extracting section 52.

First, the binary difference signal output from the comparing section 4 and the horizontal and vertical differential image data output from the edge detecting section 20 are input to the horizontal / vertical differential value density projecting section 51. As shown in FIG. 6, the horizontal / vertical differential value density projection unit 51 projects the horizontal and vertical differential image data on the Y and X axes, respectively. Since the display frame such as a blackboard is a stationary object, the density projection processing is performed on an image area in which there is no difference binary signal indicating a change in motion. Further, in order to prevent a display frame such as a blackboard from being hidden by a moving object, processing is performed so as to obtain the maximum value of density projection data obtained by density projection for a predetermined time.

The obtained density projection data is input to the projection maximum value coordinate extraction unit 52, and the maximum value coordinates (BX1, BX2), (BY1, BY) up to the second on each of the X axis and the Y axis.
2) is obtained by the imaging maximum value coordinate extraction unit 52. As a result, as shown in FIG. 6, the coordinates of the display frame such as a blackboard on the image are obtained. The obtained coordinates are output to the extracted coordinate determination unit 19. Further, for example, when the difference between the two maximum value coordinates is equal to or smaller than a predetermined value, the next maximum value (for example, the third maximum value) is obtained. When the value of the local maximum value is equal to or less than the predetermined value, processing is performed assuming that a display frame such as a blackboard does not exist on the image.

Next, the pointing rod detecting section 22 shown in FIG. 1 will be described in detail. FIG. 7 is a block diagram showing a configuration of the pointing stick detection unit shown in FIG. FIG. 8 is a diagram for explaining the operation of the pointing stick detection unit shown in FIG.

In FIG. 7, the pointing rod detection unit includes a horizontal differential value Y-axis density projection unit 71, a specified Y coordinate extraction unit 72, a horizontal differential value X-axis density projection unit 73, and a specified X coordinate extraction unit 74.

First, the binary difference signal output from the comparing section 4 and the horizontal differential image data output from the edge detecting section 20 are input to the horizontal differential value Y-axis density projecting section 71. The horizontal differential value Y-axis density projection unit 71 determines that only the horizontal differential image data is Y for the image area where the differential binarized signal exists.
Density projection is performed on the axis. The obtained density projection data on the Y axis is output to the designated Y coordinate extraction unit 72.

The designated Y coordinate extracting unit 72 obtains the Y coordinate PY of the maximum value of the input density projection data. In this process, as shown in FIG. 8, the Y coordinate of a horizontally elongated object having a large horizontal differential value, that is, the pointing rod, is obtained among the moving objects. On the other hand, if the value of the maximum value of the input density projection data is equal to or less than a predetermined value, the instruction Y coordinate extraction unit 72 determines whether there is an indicator rod for pointing an object on the image.
Or the wand is treated as if it were present but not moved.

Next, the designated Y coordinate extracting unit 72 obtains the vertical position coordinates (PY1, PY2) where the pointing bar shown in FIG.

PY1 = PY-PH / 2 PY2 = PY + PH / 2 where PH: fixed value The vertical position coordinates (PY1, PY
2) is output to the horizontal differential value X-axis density projection unit 73.

Next, the vertical position coordinates (PY1, PY
2) The difference binary signal output from the comparison unit 4 and the horizontal differential image data output from the edge detection unit 20 are input to the horizontal differential value X-axis density projection unit 73, respectively.
The horizontal differential value X-axis density projection unit 73 converts the horizontal differential image data of the area surrounded by the vertical position coordinates (PY1, PY2) where the differential binarized signal indicating the change in motion exists and the pointing bar exists. Density projection is performed on the X-axis, and the density projection data is specified.
Output to the coordinate extraction unit 74. Here, as shown in FIG. 8, a region having the maximum horizontal length is found in the density projection data, and the region is divided into two left and right regions S1 and S2 in the horizontal direction.
Divided into Comparing the cumulative values of the density projection data of each of the two divided areas S1 and S2, the cumulative value of the density projection data is larger in the area S1 where the person holding the pointing stick exists, that is, S1> S2 Therefore, the coordinates of the end of the area S2 where the accumulated value of the density projection data is small are set as the X-coordinate PX of the pointing rod. As a result, the designated Y coordinate extraction unit 72
And the coordinates (PX, PY) of the pointing stick obtained by the pointing X coordinate extracting unit 74 are output to the extracted coordinate determining unit 19.

The coordinates (PX, PY) of the above-mentioned pointing rod
May not completely match the position indicated by the pointing rod, but here, it is sufficient if the approximate position indicated by the pointing rod can be detected.

Next, the speaker determining unit 18 and the display frame detecting unit 2
1. The coordinates of the speaker among the plurality of persons, the display frame such as a blackboard, and the position pointed to by the pointer are obtained by the pointer detection unit 22 are input to the extracted coordinate determination unit 19. The extracted coordinate determination unit 19 determines which coordinate is the coordinate of the target object from the input data. This determination is made by the user of the videophone device or the video conference system including the image processing device of the present invention designating the use status in the extraction target object priority designation unit 23. For example, which target objects do you want to display preferentially?
The user specifies the content such as whether to use a display frame such as a blackboard, to indicate an object to be explained with a pointing stick, or to superimpose and display a plurality of objects. The specified content is output from the extraction target object priority specification unit 23 to the extraction coordinate determination unit 19, and the extraction coordinate determination unit 19 determines a target target object according to the specified content. Further, processing such as automatically determining the enlargement zoom magnification of the target object according to the size of the target object may be performed. By the above processing, an extraction region instruction signal for designating a region including the target object is finally output.

In the above embodiment, in order to determine a plurality of persons, the calculation of the approximate horizontal edge coordinates (Xa, Xb) of each person is performed for a plurality of independent motion regions. However, depending on the imaging display angle of an imaging device such as a CCD camera, there is a situation where a plurality of human images overlap. In such a situation, when the approximate horizontal position coordinates (Xa, Xb) of the object are obtained from the motion area and the back position of the head of the person is obtained by the above processing, the back position of the head is correctly calculated. May not be represented, and the face may not be recognized as a face even if the face recognition processing is performed on the position behind the head. Further, even if the face is correctly determined, only one of a plurality of different persons can be determined, and no other person can be determined.

Further, in order to obtain the position of an object which is a topic, the position of a blackboard or the like or the position where a pointing stick or the like points in the horizontal direction is used. Even when the object is not used, or when the pointing rod or the like is used, the detection of the pointing rod or the like cannot be performed when the object is pointed by the pointing rod or the like from a direction other than the horizontal direction. In such a case, it is not possible to determine the position of the target object that is being discussed. A second embodiment that solves the above-described problem will be described in detail below.

That is, in the second embodiment, even if a plurality of persons are photographed in an overlapping manner, each person can be recognized. FIG. 9 is a block diagram showing the configuration of the image processing apparatus according to the second embodiment of the present invention.

Referring to FIG. 9, the image processing apparatus includes an A / D
Converter 101, motion area detector 102, binary converter 1
03, first face candidate position detecting section 104, face width extracting section 10
8, a face feature extraction determination unit 109, an edge detection unit 110, and a second face candidate position detection unit 111.

Hereinafter, the operation of the image processing apparatus will be described. First, luminance information Y of a video signal input by an imaging device such as a CCD camera is input to an A / D converter 101 and converted into a digital luminance signal. The converted digital luminance signal is output to motion area detecting section 102, binary converting section 103, and second face candidate position detecting section 111.

The moving area detecting section 102 operates in the same manner as the image memory 2 and the absolute difference section 3 shown in FIG. That is, the difference between the frames of the input digital luminance signal is calculated, and the time difference data is output to the binary conversion unit 103.

[0099] Binary conversion section 103 performs the same operation as threshold value determination section 5 and comparison section 4 shown in FIG. That is, the binary conversion unit 103 determines whether the predetermined threshold value determined by the input digital video signal and the motion area detection unit 10
2, the time difference data is binarized, and the difference binarized signal is converted into a first face candidate position detecting unit 104, a face width extracting unit 108, and a second face candidate position detecting unit 111. Output to

Edge detecting section 110 performs the same operation as edge detecting section 20 shown in FIG. That is, the input digital luminance signal is differentiated in the vertical and horizontal directions of the image to obtain vertical differential image data and horizontal differential image data, which are output to the face width extraction unit 108 and the face feature extraction determination unit 109.

The first face candidate position detecting section 104 includes a horizontal moving area extracting section 105, a moving area selecting section 106, and a top extracting section 107. The horizontal moving area extraction unit 105
The operation is the same as that of the X-axis density projection unit 6, continuous region peak detection units 7, 10, continuous region independent comparison units 8, 11, time direction accumulation unit 9, and object region width extraction unit 12 shown in FIG. The moving area selection unit 106 performs the same operation as the calculation object selection unit 13 shown in FIG. The parietal extraction unit 107 determines the Y value shown in FIG.
The same operation as the axis density projection unit 14 and the maximum Y-axis projection coordinate extraction unit 15 is performed.

Therefore, the horizontal moving area extraction unit 10
5 obtains the horizontal position coordinates of the moving object for each moving object by cumulatively adding the input difference binary signal spatially in time and in the horizontal direction. Moving area selector 1
In step 06, the horizontal position coordinates (Xa, Xb) of one moving object to be processed after this processing are selected from the obtained horizontal position coordinates of the plurality of moving objects, and the top extraction unit 107 and the face width extraction unit Output to 108. Head extraction unit 107
Are the horizontal position coordinates (Xa, X
The binarized difference signal is cumulatively added in the vertical direction to the inclusive section (horizontal coordinate = Xa to Xb) of b), and the topmost position Ytop of the cumulative value in the vertical direction, that is, the top coordinate of the person is obtained.

Next, the second face candidate position detecting section 111 will be described in detail. The second face candidate position detecting unit 111
/ D converter 112, hue calculation unit 113, comparison unit 114,
A constant value generator 115 and a face hue position calculator 116 are included.

The A / D converter 112 receives color information C of a video signal input by an image pickup device such as a CCD camera and converts it into a digital color signal.

The digital luminance signal and the digital color signal are input to the hue calculating section 113, and the hue calculating section 113 calculates the hue value of the image. The calculated hue value is supplied to the comparing unit 11
4 is output.

A predetermined constant value corresponding to the hue value is input from the constant value generation unit 115 to the comparison unit 114. The comparing unit 114 compares the input hue value and the constant value, and outputs a hue region signal indicating a region having a hue value corresponding to the constant value to the face hue position calculating unit 116. For example, when determining the position of a human face, the constant value generation unit 115 outputs a constant value corresponding to the skin color of the face as a hue constant value. next,
The constant value representing the skin color is compared with the hue value, an area having the hue value of the skin color is obtained by the comparing section 114, and a hue area signal indicating the area having the hue value of the skin color is obtained by the face hue position calculating section 116.
Output to

Based on the input hue area signal and differential binarized signal, face hue position calculating section 116 determines a target hue area and a differential binarized signal area, that is, a skin color area with motion. Ask. The face hue position calculation unit 116 calculates
The horizontal position coordinates (X
a ′, Xb ′) and parietal coordinates (Ytop ′),
Output to the face width extraction unit 108.

With the above operation, it is possible to obtain the horizontal position coordinates and the top coordinates of a region having a specific hue, for example, a hue value of a flesh color. Even when a plurality of persons overlap, the positions of the faces of a plurality of persons can be obtained. Can be specified. For example, when three persons overlap and there are three skin color areas, that is, face areas, the three kinds of horizontal position coordinates (Xa ′, Xb ′) and top coordinates (Ytop ′) are determined by the face width extraction unit 108. Output to

In the face width extracting unit 108, the binary converting unit 103
Binarized signal and edge detector 11 output from
The vertical differential image data output from 0 is input,
Further, the horizontal position coordinates (Xa, Xb) and the top coordinate (Ytop) of the moving object output from the first face candidate position detection unit 104, and the second face candidate position detection unit 111
Horizontal position coordinates (X
a ′, Xb ′) and the parietal coordinates (Ytop ′) are input. In the face width extraction unit 108, the first face candidate position detection unit 1
04 and the top coordinates output from the second face candidate position detection unit 111 are selectively used.
When a plurality of persons overlap, the position coordinates and the top coordinates output from the second face candidate position detection unit 111 are preferentially used. The operation of the face width extraction unit 108 operates in the same manner as the face width extraction unit 16 shown in FIG. 1, and outputs the X coordinate (Lcor, Rcor) of the face width of the person from the input data to the face feature extraction determination unit 109. Output.

The face feature extraction judging section 109 operates in the same manner as the face feature extraction judging section 17 shown in FIG. 1 to judge whether or not it is the head of a person, and outputs the judgment result to the speaker judging section 18 shown in FIG. Output to Each block after the speaker determination unit 18 is the same as in FIG. 1 and each operation is the same as in the first embodiment shown in FIG.

With the above operation, in the second embodiment, the second
A region corresponding to the hue, such as a skin color region, can be specified by using the face candidate position detection unit 111 and can be output to the face width extraction unit 108. Therefore, even when a plurality of people overlap, the position of each person can be specified. It is possible to do. Next, another configuration example of the pointing stick detection unit 22 illustrated in FIG. 1 will be described. FIG. 10 is a block diagram illustrating the configuration of the pointing stick detection unit of the image processing device according to the third embodiment of the present invention.

Referring to FIG. 10, the pointing rod detection unit includes a narrow area calculation unit 130, an area expansion unit 133, and a small area removal unit 1.
34, a feature amount detection unit 135, a pointing stick determination unit 136, and a pointing position detection unit 137. Further, the narrow area calculation unit 130
Includes a contraction / expansion part 131 and a difference part 132.

The contraction / expansion unit 131 receives a differential binary signal indicating a moving area in the input image from the comparison unit 4 shown in FIG. The contraction / expansion unit 131 calculates the input difference 2
The binarized signal is eroded a predetermined number of times to remove an area having a small width from the differential binarized signal, and to perform an expansion process the same number of times as the number of erosion processes and return an area other than the deleted area to the original width. . The difference binarized signal from which the narrow area has been removed by the contraction / expansion unit 131 and the original difference binarized signal are input to the difference unit 132. The difference unit 132 calculates the difference between the two,
Extract narrow areas.

A signal indicating the extracted narrow area is input to the area expanding section 133. The region expanding unit 133 obtains connection information of the eight neighboring regions of the extracted region, and combines a plurality of regions that are adjacent in the vicinity based on the eight neighboring connection information into one region.

The small area removing section 134 includes an area expanding section 133
By extracting only the area having an area equal to or more than a certain value from the areas combined into one area, only the area having a motion and having a small width and an area equal to or more than a certain value is extracted.

The feature amount detection unit 135 is provided with the small area removal unit 13
The following feature amounts are obtained for the area obtained in step 4. FIG. 11 is a diagram showing the coordinates of the pointer, and FIG.
FIG. 2 is a diagram showing characteristic amounts of the pointing stick shown in FIG. In other words, the feature amount detection unit 135 determines the maximum feature length L of the pointing stick shown in FIGS.
B, the maximum vertical width W, and the main axis direction θ are determined, and the pointing rod determination unit 1
36.

The pointing bar determination unit 136 determines whether or not the ratio between the determined maximum graphic length L and the maximum vertical width W of the region is equal to or more than a certain value, in other words, based on each input characteristic amount. It is determined whether or not the area is an elongated area such as a pointing stick. When the obtained area is an elongated area such as an indicating rod or the like, the pointing rod determination unit 136 determines the area of the difference binarized signal with respect to the area centered on the end point (A, B) of the maximum graphic length of the area. Check the value and find two end points (A, B)
Is determined, and an area B having a small area value, that is, an end point B at which a human does not have the pointing bar is obtained, and the coordinates are output as the coordinates (PbX, PbY) of the pointing bar. The output coordinates (PbX, PbY) of the pointer are output to the extracted coordinate determination unit 19 shown in FIG. 1 and are processed in the same manner as in the first embodiment.
By the above processing, the pointing rod detection unit shown in FIG. 10 can specify the position of the pointing rod even when the target object is pointed by an elongated object such as the pointing rod from any direction other than the horizontal direction. As a result, the position of the object can be specified.

Next, an image processing apparatus according to a fourth embodiment of the present invention will be described with reference to the drawings. In the fourth embodiment, a center object detection unit that determines an object at the center of the screen using horizontal and vertical differential image data is provided instead of the pointing bar detection unit 22 of the first embodiment shown in FIG. Things. FIG. 13 is a block diagram illustrating a configuration of a central object detection unit of an image processing device according to a fourth embodiment of the present invention. 4th
The second embodiment is the same as the first embodiment except that a center object detection unit described below is provided instead of the pointing stick detection unit 22 shown in FIG. Only the object detection unit will be described in detail.

Referring to FIG. 13, the central object detecting section comprises:
A horizontal / vertical projection unit 141 and a central object determination unit 142 are included.

The horizontal / vertical projection section 141 receives the horizontal / vertical differential image data output from the edge detection section 20 shown in FIG. FIG. 15 is a diagram for explaining the operation of the central object detection unit shown in FIG. The horizontal / vertical projection unit 141 outputs the differential image data in the horizontal and vertical directions with respect to an area surrounded by a horizontal central area CX and a lower area CY from the vertical center of the input image shown in FIG. Find the cumulative differential projection value. As shown in FIG. 15, the cumulative differential projection values in the horizontal and vertical directions are the area A having the same cumulative differential projection value in the horizontal and vertical directions and the area having the same cumulative differential projection value around the area A. B and C are present. Also, here
Regions B and C have approximately equal cumulative differential projection values.
The area A is an area corresponding to an object near the center of the screen, and the areas B and C are areas corresponding to a conference table, a desk, and the like. Therefore, by determining whether there is an area having different differential data among areas having similar differential data, such as a conference table or a desk, it is possible to determine where the object is.

Therefore, the central object determining section 142 uses the cumulative differential projection value cumulatively projected by the horizontal / vertical projection section 141 to determine whether there is an area A having a different cumulative differential projection value from the areas B and C around it. No, in other words, it is determined whether there is an area having different differential data, that is, an object, in an area having equivalent differential data such as a conference table or a desk. The central object determining unit 142 determines the coordinates (PX, P
Y) is an extracted coordinate determining unit 1 shown in FIG.
9 is output. Through the above processing, even when the pointing rod or the like is not used for the object, the position of the object can be specified using the horizontal and vertical differential image data.

Next, an image processing apparatus according to a fifth embodiment of the present invention will be described. In the fifth embodiment, an object is detected by using a difference between hue values. FIG. 14 is a block diagram showing the configuration of the central object detection unit of the image processing device according to the fifth embodiment of the present invention. In the fifth embodiment, FIG.
Is the same as that of the first embodiment except that a center object detection unit using a difference in hue amount is used instead of the pointing rod detection unit 22 of the first embodiment shown in FIG. Only the central object detection unit will be described in detail.

Referring to FIG. 14, horizontal / vertical projection unit 143
The hue value output from the hue calculation unit 113 shown in FIG. The horizontal / vertical projection unit 143 calculates the cumulative hue projection values of the hue values in the horizontal and vertical directions, similarly to the horizontal / vertical projection unit 141 shown in FIG. The central object determining unit 144 operates in the same manner as the central object determining unit 142 illustrated in FIG. 13, and calculates a different hue from an obtained cumulative hue projection value into an area having an equivalent hue value such as a conference table or a desk. It is determined whether there is an area having a value, that is, an object. If there is an object, the position coordinates (PX, P
Y) is output to the extracted coordinate determining unit 19 shown in FIG.

By the above processing, even when the pointing rod or the like is not used for the object, the position of the object can be specified by using the difference in the hue value.

The above embodiments can be arbitrarily combined with each other, in which case the respective functions and effects can be obtained.

According to the above-described processing, the image processing apparatus according to the present invention first determines the contents of the transferred image when the videophone or the video conference system is used. By determining the positions of multiple people using features of the face and the face, and determining the person who is currently speaking from multiple people from the movement of the lips, the person who is currently speaking from multiple people in a meeting etc. It is possible to output an extraction area instruction signal for automatically instructing an image that has been set as an image, and by using this extraction area instruction signal, an image centering on the speaker can be automatically made a transfer image. .

Further, image data captured by an imaging device such as a CCD camera can be differentiated, and the position of an object such as a display frame such as a blackboard and a long and thin stick can be automatically obtained from the differentiated image data. In a state other than when speaking at a meeting or the like, that is, when a document or drawing is described on a blackboard or the like or an article of a topical product is described, a display frame or a pointing stick that is being described. It is possible to automatically set an image centered on the position indicated by as the transfer image.

Further, the approximate position of the person is obtained from the image obtained by the imaging device such as a CCD camera using the hue information of the face, and the feature of the pattern of the face of the person is used for the position. It is possible to obtain the positions of a plurality of persons even when the persons overlap. further,
In order to detect the position of a topical object, it is possible to detect a moving part with an imaging device such as a CCD camera, regardless of the direction of an object such as an elongated pointing rod in any direction. By utilizing the difference between the differential value and the hue value of the conference table, desk, or the like and the object for the judgment of the object, each object can be detected.

Further, the user sets in advance the combination of the selected transfer images, the priority order, the presence / absence of enlarged display, and the like according to the content of the conference, so that the user can set the desired image. Automatic selection of images, combination, and automatic switching of enlargement ratio without manual instructions.
This image can be used as a transfer image.

[0108]

According to the image processing apparatus of the present invention, since the positions of a plurality of moving objects can be detected, a desired object can be specified.

In the image processing apparatus according to the second aspect,
Since the position of the speaker can be detected, it is possible to specify a desired target object.

In the image processing apparatus according to the third aspect,
Since the position of the stationary object can be detected, a desired target object can be specified.

In the image processing apparatus according to the fourth aspect,
Since the position of an elongated moving object can be detected, a desired target object can be specified.

In the image processing apparatus according to the fifth aspect,
A predetermined hue region is specified using the hue value, a position of a predetermined target object corresponding to the hue region can be specified, and a desired target object can be specified.

In the image processing apparatus according to the sixth aspect,
An elongated region having a narrow width and an area equal to or larger than a predetermined value can be calculated regardless of the direction of the target object, so that a desired target object can be specified.

In the image processing apparatus according to the seventh aspect,
Since the position of the target object is specified using the differential image data, a desired target object can be specified even when the pointing rod or the like is not used.

In the image processing apparatus according to the eighth aspect,
Since the position of the target object is specified using the difference in hue value, it is possible to specify a desired target object even when the pointing rod or the like is not used.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram for explaining processing from density projection processing to object area width extraction processing.

FIG. 3 is a diagram illustrating a speaker determination process.

FIG. 4 is a block diagram illustrating a configuration of a speaker determination unit illustrated in FIG. 1;

FIG. 5 is a block diagram illustrating a configuration of a display frame detection unit illustrated in FIG. 1;

FIG. 6 is a diagram for explaining the operation of the display frame detection unit shown in FIG.

FIG. 7 is a block diagram illustrating a configuration of a pointing stick detection unit illustrated in FIG. 1;

FIG. 8 is a diagram for explaining an operation of the pointing stick detection unit shown in FIG. 1;

FIG. 9 is a block diagram illustrating a configuration of an image processing apparatus according to a second embodiment of the present invention.

FIG. 10 is a block diagram illustrating a configuration of a pointing rod detection unit of an image processing apparatus according to a third embodiment of the present invention.

FIG. 11 is a diagram showing coordinates of a pointing stick.

FIG. 12 is a diagram showing characteristic amounts of the pointing stick shown in FIG. 11;

FIG. 13 is a block diagram illustrating a configuration of a central object detection unit of an image processing apparatus according to a fourth embodiment of the present invention.

FIG. 14 is a block diagram illustrating a configuration of a central object detection unit of an image processing apparatus according to a fifth embodiment of the present invention.

FIG. 15 is a diagram for explaining the operation of the central target object detection unit shown in FIG.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 A / D converter 2 Image memory 3 Absolute difference part 4 Comparison part 5 Threshold determination part 6 X-axis density projection part 7, 10 Continuous area peak detection part 8, 11 Continuous area independent comparison part 9 Time direction accumulation part 12 Object area width extraction unit 13 Calculated object selection unit 14 Y-axis density projection unit 15 Maximum Y-axis projection coordinate extraction unit 16 Face width extraction unit 17 Face feature extraction determination unit 18 Speaker determination unit 19 Extraction coordinate determination unit 20 Edge detection unit 21 Display frame detecting unit 22 Pointer detecting unit 23 Extraction object priority order specifying unit

Continuation of front page (56) References JP-A-2-236787 (JP, A) JP-A-5-205157 (JP, A) JP-A-2-81590 (JP, A) JP-A-62-46384 (JP) JP-A-1-314385 (JP, A) JP-A-61-134884 (JP, A) JP-A-62-84391 (JP, A) JP-A-3-182189 (JP, A) 2-108167 (JP, A) JP-A-5-87631 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T ⁷ /00-7/60 G06T 1/00 G06T 5 / 00

Claims (8)

(57) [Claims] An image data is analyzed to detect a predetermined object.
An image processing apparatus that outputs the time difference data of continuous image data including a plurality of moving objects; an addition unit that cumulatively adds the time difference data; and a time difference that is cumulatively added by the addition unit. the data,
For each of the plurality of moving objects, the cumulatively added time difference
Variable binarization means for binarizing with a threshold value for each of the plurality of moving objects calculated based on data; and the plurality of moving objects based on data binarized by the binarization means. An image processing apparatus comprising: a detecting unit that detects a position of the image. 2. The image data includes data representing a person.
The image processing device includes image data binarized by the binarization unit ,
Data representing the contour of a person's face and a ring of elements in the face
First specifying means for specifying a first area including a face of a person based on at least one of the data representing the gusset; and second specifying for specifying a second area including a lip of the person from the first area Means, detecting the amount of movement of the lips in the second area, and determining the position of the speaker
The method according to claim 1 , further comprising: a third specifying unit configured to detect .
The image processing apparatus. 3. The image data representing a stationary object.
Includes data, the image processing apparatus includes a differential means for differentiating said image data to create the differentiated image data, and a projection means for generating projection data by projecting the differentiated image data in a predetermined direction, the projection data The image processing apparatus according to claim 1 , further comprising : a stationary object detecting unit configured to detect a position of the stationary object based on the extreme value of the stationary object. 4. The image data representing an elongated object.
Comprises over data, the image processing apparatus includes a motion area detection means for detecting a movement area including the elongated moving object from the image data, a differentiating means for generating differential image data by differentiating said image data, said Extraction from the motion area based on the differential image data
Projecting means for projecting the density data of the obtained area in a predetermined direction to create projection data; and moving object detecting means for detecting a position of the elongated moving object based on an extreme value of the projection data, further comprising : Item 1
An image processing apparatus as described in the above . 5. An image processing apparatus comprising : a motion area detecting means for detecting a motion area in an image composed of continuous image data using time difference data of luminance data in the continuous image data; Binarizing means for binarizing luminance data corresponding to the image data with a predetermined threshold value and outputting binarized data; and a hue value calculating means for calculating a hue value based on the luminance data and the color data of the image data. And comparing the hue value with a predetermined constant value to determine whether the predetermined hue exists.
Further comprising, claims and hue area calculating means for calculating hue region, the target position calculating means for calculating a position of a predetermined object in the color area and the binarized data the color area based on the 2. The image processing device according to 1. 6. An image processing apparatus, comprising : a motion area detecting means for detecting a motion area in an image composed of the continuous image data using time difference data of the continuous image data; and a luminance corresponding to the motion area. Binarizing means for binarizing the data with a predetermined threshold value and outputting binarized data; performing erosion and dilation processing on the binarized data; 2 after expansion
A narrow area calculating means for calculating a narrow area of the binarized data based on a difference from the binarized data; and obtaining neighborhood area connection information for the narrow area; The image processing apparatus according to claim 1 , further comprising : an elongated area calculating unit configured to calculate an elongated area that is narrow and has an area equal to or larger than a predetermined value. 7. An image processing apparatus , comprising: differentiating means for differentiating image data and calculating differential image data in horizontal and vertical directions; and the horizontal and vertical directions with respect to an area near the center of an image composed of the image data. Horizontal and vertical projection means for projecting differential image data in the vertical and horizontal axes, respectively, and outputting horizontal and vertical projection data; and a predetermined object near the center of the image using the horizontal and vertical projection data. of determining a determination unit configured to determine whether there is a by the determining means near the center of the image
Extraction for extracting image data corresponding to a predetermined target object
And means, the image processing apparatus according to claim 1. 8. An image processing apparatus comprising : hue value calculating means for calculating hue values in the horizontal and vertical directions from color data of image data; Horizontal and vertical hue projection means for projecting hue values in the vertical direction onto the horizontal axis and the vertical axis, respectively, and outputting horizontal and vertical hue projection data; and and determination means for determining whether there is an object, to be near the center of the image by the determination means determines
Extraction to extract image data corresponding to the specified target object
And means, the image processing apparatus according to claim 1.