CN101547307B - Image processing device and program - Google Patents
Image processing device and program Download PDFInfo
- Publication number
- CN101547307B CN101547307B CN2009101268897A CN200910126889A CN101547307B CN 101547307 B CN101547307 B CN 101547307B CN 2009101268897 A CN2009101268897 A CN 2009101268897A CN 200910126889 A CN200910126889 A CN 200910126889A CN 101547307 B CN101547307 B CN 101547307B
- Authority
- CN
- China
- Prior art keywords
- sight line
- image processing
- facial
- personages
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The present invention provides an image processing device and method, capable of reliably picking-up field wherein most persons of a plurality of persons as picked-up bodies look same object in the image. The image processing device has a face detection section detecting faces of a plurality of persons included in the image according to image mode in the image signal; a sight line estimation section for estimating the sight line of the persons of faces detected by the face detection section; a distance calculation section respectively calculating distances of persons estimated by the sight line estimation section; and an interleaving determination section determining whether vision line of the persons is interleaved by using the estimated result and the calculated result.
Description
Technical field
The present invention relates to picture signal is carried out the image processing apparatus and the image processing method of predetermined processing.
Background technology
In the past, in having the image processing apparatus such as digital camera of camera function, known have a following technology: in subject was under people's the situation, detected this person's face, was focused in the position of this detected face.And in recent years, it is also known for following technology: automatically identification shows smiling face as the people of subject and waits the moment of the expression that is suitable for photographing to photograph.
Disclosed following technology: under this situation, when a plurality of personages are photographed as object, judge the position of each personage's face orientation and black eye ball, extract the image (for example with reference to patent documentation 1) of camera lines of sight thus.And, following technology is also disclosed: infer the direction of visual lines of detected personage from image, extract the feature of this direction of visual lines, judge this personage's action intention (for example with reference to patent documentation 2) thus.
[patent documentation 1] TOHKEMY 2007-96440 communique
[patent documentation 2] TOHKEMY 2007-6427 communique
Under carrying out the situation of a plurality of personages as the photography of subject, generally the state that the full complement is in camera lines of sight is taken, and still, in addition also has the scene that is suitable for photographing.For example a plurality of personages' sight line is watched the scene of image processing apparatus same object thing in addition etc. attentively, in image, produce unified sense, and, the moment that can exchange each other can also be caught by the personage that the behavior of watching the same object thing attentively is spied on, so, be suitable for object as photography.
But, in above-mentioned prior art, can't extract a plurality of personages and watch the image processing apparatus such scene of object in addition simultaneously attentively.Particularly, in the technology that above-mentioned patent documentation 1 is put down in writing, only judge a plurality of personages' camera lines of sight, can't judge whether a plurality of personages' sight line concentrates on camera object in addition.And, in the technology that above-mentioned patent documentation 2 is put down in writing, only infer each personage's face orientation and sight line towards, can't consider the processing of the sight line relations each other of a plurality of personages in the image.
Therefore, in existing image processing apparatus, in order to extract the scene of watching image processing apparatus object in addition as a plurality of personages of subject simultaneously attentively, cameraman self has to judge by visual, under the numerous of subject or situation that subject is far, be difficult to extract this scene.
Summary of the invention
The present invention In view of the foregoing finishes, its purpose is, following image processing apparatus and image processing method is provided: it can extract the scene that most of personages watch the same object thing attentively in comprising the image of a plurality of personages as subject reliably.
In order to solve above-mentioned problem and to reach purpose, image processing apparatus of the present invention is characterised in that this image processing apparatus has: facial test section, and it detects a plurality of personages' that comprised in the described picture signal face according to the image model in the picture signal; Sight line is inferred portion, and it infers the sight line that is detected facial described a plurality of personages by described facial test section; Distance calculation portion, it calculates the distance of inferring out described a plurality of personages of sight line apart from by the described sight line portion of inferring respectively; And the intersection detection unit, it uses described sight line to infer the result of calculation of inferring result and described distance calculation portion of portion, judges whether described a plurality of personages' sight line intersects.
And, image processing apparatus of the present invention is characterised in that in foregoing invention, this image processing apparatus has display part, this display part shows the view data corresponding with described picture signal, the result of determination of described display part described intersection detection unit of overlapping demonstration on described view data.
And image processing apparatus of the present invention is characterised in that in foregoing invention, the overlapping demonstration on described view data of described display part is used for the personage's of the mutual sight line intersection of definite described a plurality of personages information.
And image processing apparatus of the present invention is characterised in that in foregoing invention, this image processing apparatus has cutting portion, and this cutting portion carries out the cutting of described view data according to the result of determination of described intersection detection unit.
And image processing apparatus of the present invention is characterised in that, in foregoing invention, the described sight line portion of inferring has: facial direction calculating portion, its calculate by the detected face of described facial test section towards; And the direction of visual lines calculating part, its use the described face that calculates by described facial direction calculating portion towards, and calculate pupil that this face comprises towards.
And image processing apparatus of the present invention is characterised in that, in foregoing invention, described intersection detection unit is a benchmark with the maximum personage of distance who is calculated by described distance calculation portion, judges whether this benchmark personage's sight line and other personages' sight line intersect.
And, image processing apparatus of the present invention is characterised in that, in foregoing invention, described intersection detection unit is got rid of in the situation of intersecting with the distance of this image processing apparatus position more farther than described benchmark personage described benchmark personage's sight line and described other personages' sight line from judge object.
And image processing apparatus of the present invention is characterised in that in foregoing invention, this image processing apparatus has: image pickup part, and it is made a video recording to the subject in the area of visual field of regulation, and generates picture signal; Input part, its input are used to indicate the photography index signal by the photography of the captured subject of described image pickup part; And control part, it is according to the control of carrying out from the described photography index signal of described input part input photographing to by the captured subject of described image pickup part.
And image processing apparatus of the present invention is characterised in that, in foregoing invention, described control part carries out the control that exposes and/or focus by the detected face of described facial test section.
And image processing apparatus of the present invention is characterised in that in foregoing invention, described control part carries out the control to being photographed by the captured subject of described image pickup part automatically according to the result of determination of described intersection detection unit.
Image processing method of the present invention is characterised in that, this image processing method is the image processing method that picture signal is carried out the image processing apparatus of predetermined processing, may further comprise the steps: face detection step, in this step, according to the image model in the described picture signal, detect a plurality of personages' that comprised in the described picture signal face; Sight line is inferred step, in this step, infers the sight line that detects facial described a plurality of personages in described face detection step, is stored to the facial information storage part that is used for storing facial information; The distance calculation step in this step, is calculated the distance of inferring the described a plurality of personages that infer out sight line in the step apart from described sight line respectively, is stored in the described facial information storage part; And intersection determination step, in this step, read described sight line and infer the result of calculation of inferring result and described distance calculation step of step from described facial information storage part, that uses that this reads infers result and result of calculation, judges whether described a plurality of personages' sight line intersects.
According to the present invention, infer the sight line of a plurality of personages in the picture signal, calculate respectively apart from the distance of inferring out a plurality of personages of sight line, use these sight lines to infer result and distance calculation result, whether the sight line of judging a plurality of personages intersects, so, can extract the scene that most of personages watch the same object thing attentively in comprising the image of a plurality of personages as subject reliably.
Description of drawings
Fig. 1 is the figure of the structure of image processing apparatus (camera head) that one embodiment of the present invention is shown.
Fig. 2 is the figure of example of the tupe of image processing apparatus (camera head) that one embodiment of the present invention is shown.
Fig. 3 be image processing apparatus (camera head) that one embodiment of the present invention is shown comprise the flow chart of sight line determination processing in the flow process of interior processing.
Fig. 4 is the flow chart that the summary of sight line determination processing is shown.
Fig. 5 is the figure that the structure of facial zone is shown.
Fig. 6 is the figure that the summary of facial model of ellipse (horizontal direction) is shown.
Fig. 7 is the figure that the summary of facial model of ellipse (vertical direction) is shown.
Fig. 8 is the figure that the summary of eyeball phantom is shown.
Fig. 9 is the schematic diagram of observing from the top of image processing apparatus as the relation between the target that a plurality of personage and these a plurality of personages watched attentively of subject.
Figure 10 is the figure of summary that the distance calculation of subject is shown.
Figure 11 is the schematic diagram of observing from the side of image processing apparatus as the relation between the target that a plurality of personage and these a plurality of personages watched attentively of subject.
Figure 12 is the figure that the demonstration example (the 1st example) of OK demonstration is shown.
Figure 13 is the figure that the demonstration example (the 2nd example) of OK demonstration is shown.
Figure 14 is the figure that the demonstration example (the 3rd example) of OK demonstration is shown.
Figure 15 is the figure that cutting result's demonstration example is shown.
Figure 16 is the figure of the demonstration example in the display part after the sight line determination processing of image processing apparatus (camera head) that other execution modes of the present invention are shown.
Label declaration
1: camera head; 2: image pickup part; 3: sensor part; 4: image processing part; 5: display part; 6: display control unit; 7: compressed and decompressed portion; 8: the recording medium interface; 9: input part; 10: audio output unit; 11: the fill-in light projection unit; 12: control part; 13: storage part; 14: lens driving portion; 15: the aperture drive division; 21: phtographic lens; 22: aperture; 23: imaging apparatus; 24: analog portion; The 25:A/D converter section; 31: facial zone; 32: the facial zone center; 33: the face organ center; 41: facial test section 42: sight line is inferred portion; 43: distance calculation portion; 44: the intersection detection unit; 45: cutting portion; 51,52,53,55: display image; 52a: sight line concentrates the personage to determine symbol 52b, 53a: sight line does not concentrate the personage to determine symbol; 54: cutting is display image 54a as a result: the clipping region; 100: recording medium; 121: clock and watch; 131: program storage part; 132: the facial information storage part; 200,201,202: face; 300,500: the perspective plane; 400: eyeball 401: pupil; 421: facial direction calculating portion; 422: the direction of visual lines calculating part; A1, A2, A11～A13, A21～A25: personage; CP: intersection point; T: target.
Embodiment
Below, be used to implement optimal way of the present invention (being called " execution mode " later on) with reference to description of drawings.
Fig. 1 is that the image processing apparatus that an embodiment of the invention schematically are shown is the figure of the structure of camera head 1.Camera head 1 has: image pickup part 2, and it is made a video recording to the subject in the regulation area of visual field, and generates picture signal; Sensor part 3, it detects the state of the area of visual field of image pickup part 2; Image processing part 4, its picture signal to image pickup part 2 outputs is implemented image processing; Display part 5, its demonstration comprise view data by image processing part 4 output in interior information; Display control unit 6, its demonstration to display part 5 is controlled; Compressed and decompressed 7, it carries out the compression and decompression of view data; Recording medium interface 8, it is to from being externally mounted to the information such as recording mediums such as storage card 100 recording image datas on the camera head 1, on the other hand, the information of read-out recording medium 100 records; Input part 9, the operation index signal of its input camera head 1; Audio output unit 10, it comprises the voice output of warning in interior various information; Fill-in light projection unit 11, it throws fill-in light to camera watch region; Control part 12, its action to camera head 1 is controlled; And storage part 13, the various information that its storage is relevant with the action of camera head 1.
Image processing part 4 has: facial test section 41, and it is according to the image model of picture signal, the personage's who is comprised in the detected image signal face; Sight line is inferred portion 42, and it detects at facial test section 41 under a plurality of personages' the situation of face, the sight line of inferring each personage; Distance calculation portion 43, it calculates respectively by the distance sight line and infers the distance that a plurality of personages of sight line infer out in portion 42; Intersect detection unit 44, it uses sight line to infer the result of calculation of inferring result and distance calculation portion 43 of portion 42, judges whether at least a portion sight line in a plurality of personages' the sight line intersects; And cutting portion 45, its result who judges according to intersection detection unit 44 carries out the cutting of view data.Wherein, sight line is inferred portion 42 and is had: facial direction calculating portion 421, its calculate by facial test section 41 detected faces towards (facial direction); And direction of visual lines calculating part 422, its use face that facial direction calculating portion 421 calculates towards, calculate pupil that this face comprises towards.
Input part 9 has: the power knob of camera head 1, provide shooting indication shutter release button, carry out the exercises pattern that camera head 1 has switching mode switch button and comprise the reproduction of indicating image data and editor's control button etc.
Control part 12 has the clock and watch 121 of judging photography time on date etc., according to the testing result of sensor part 3 with by the operation index signal of input part 9 inputs, the action of camera head 1 is controlled.Control part 12 uses MPU to realize, is connected with each structural element of controlling object via bus.
Fig. 2 is the figure that the part of the pattern that can set in camera head 1 is shown.In Fig. 2, be that the center is put down in writing with camera head 1 distinctive pattern, omit record about the pattern identical with the pattern that can in existing camera head, set.Be set at camera head 1 under the situation of photograph mode, can further set sight line judges photograph mode and photographs with sight line judgement cutting pattern, judge in the photograph mode in sight line, the sight line that is used for a plurality of personages that the process decision chart picture comprised is intersected the sight line of situation and is judged, photograph then, judge with sight line in the cutting pattern in photography, after photographing usually, carry out that sight line is judged and based on the cutting of the image of sight line result of determination.Wherein, be set at sight line at camera head 1 and judge under the situation of photograph mode, can further set the one or more patterns in cutting pattern, burst mode, the automatic photography pattern.On the other hand, be set at camera head 1 under the situation of reproduction mode, can further set reproduction and judge the cutting pattern with sight line, in reproducing with sight line judgement cutting pattern, when reproducing the view data that recording medium 100 write down, carry out sight line and judge, carry out the cutting of image then according to this result of determination.
Fig. 3 illustrates the flow chart that the camera head 1 with above structure comprises the handling process of sight line determination processing when interior action.
At first, illustrate that camera head 1 is set at the situation of photograph mode (step S1: photograph mode).Be set at camera head 1 under the situation of photograph mode, display part 5 shows and the corresponding view data of exporting from image pickup part 2 of picture signal in real time according to the control of display control unit 6.Below, will be called " browse graph picture " by the view data that display part 5 shows in real time.Judge photograph mode (step S2: be) if camera head 1 is set at sight line, then 1 pair of browse graph of camera head looks like to carry out sight line determination processing (step S3).
Fig. 4 is the flow chart that the summary of the sight line determination processing that camera head 1 carries out is shown.At first, facial test section 41 detects the face (step S31) that is present in the personage in the browse graph picture that is shown by display part 5.When carrying out this face detection, can use existing method (for example with reference to P.Violaand M.Jones, " Rapid Object Detection using a Boosted Cascade of SimpleFeatures ", Proc.ofCVPR (2001) .).Control part 12 carries out the control by the focusing of the position of facial test section 41 detected faces.
Detect in the browse graph picture at facial test section 41 under the situation of a plurality of faces (step S32: be), camera head 1 enters step S33.On the other hand, do not detect in the browse graph picture at facial test section 41 that (step S32: not), camera head 1 enters step S4 under the situation of a plurality of faces.
In step S33, sight line is inferred portion 42 and is carried out inferring by the sight line of the direction of visual lines of facial test section 41 detected a plurality of faces at each face and infer circular treatment (step S33).In this circular treatment, at first, facial direction calculating portion 421 calculates facial towards (facial direction) (step S331).Face direction calculating portion 421 obtains the image (face-image) of facial test section 41 detected facial parts, thereby implement processing such as difference background and area of skin color extraction by the face-image that this is obtained and extract facial zone, then, use the facial zone of this extraction to calculate facial direction.Fig. 5 is the figure that the example of the facial zone that facial direction calculating portion 421 extracts is shown.Facial zone 31 shown in this figure is rectangular areas external with people's head.Extract the center (facial zone center 32) that facial zones 31 calculate in the facial direction calculating portion 421 of facial zone 31, and, face organ center 33 calculated by the position of detecting face organs such as eyebrow, the eye tip, nose.Then, facial direction calculating portion 421 uses facial zone center 32 and face organ centers 33, comes facial direction on the calculated level face respectively and the facial direction on the vertical plane.
The figure of the summary of the facial model of ellipse that Fig. 6 uses when being the facial direction that illustrates on the calculated level face.Face 200 shown in this figure is an ellipsoid, the horizontal profile by ellipsoidal center shown in Figure 6.Facial 200 optical axis directions with respect to phtographic lens 21 (above-below direction of Fig. 6) are with angle Φ
FaceTowards horizontal right.Perspective plane 300 is corresponding to facial zone 31.And, some O '
FaceCorresponding to the facial zone center 32 on the horizontal plane of Fig. 6, some C '
FaceCorresponding to the face organ center 33 on the horizontal plane of Fig. 6.
In Fig. 6, for the face angle Φ of the facial direction of expression facial 200 on horizontal plane
Face, the width W of the horizontal direction of the facial zone 31 in the use perspective plane
Face, (the some O of the facial zone center 32 in the perspective plane 300
Face) and face organ center 33 (some C) between distance c
Face, and facial width (=2a) with the length of facial depth direction (=2b) ratio k (facial oval than), provide by following formula (1).
[numerical expression 1]
Fig. 7 is the figure of the summary of the facial model of ellipse used when the facial direction of calculating on the vertical plane is shown.Shown in Figure 7 is the vertical section at the center of face 200 by ellipsoid.Facial 200 optical axis directions with respect to phtographic lens 21 (left and right directions of Fig. 7) are with angle
Towards the top.In Fig. 7, for the face angle of the facial direction on the vertical plane of expression facial 200
Use the height h of the vertical direction of the facial zone 31 in the perspective plane 300
Face, (the some O of the facial zone center 32 in the perspective plane 300
Face) and face organ center 33 (some C) between distance c '
Face, and vertical facial dimension (=2d) (=2e) ratio k ' is provided by following formula (2) with the length of facial depth direction.
[numerical expression 2]
Then, direction of visual lines calculating part 422 uses the face angle Φ on the horizontal plane that calculates in step S331
FaceWith the face angle on the vertical plane
Calculate direction of visual lines (step S332).Direction of visual lines calculating part 422 detects pupil by pattern matching existing methods such as (pattern matching) from facial zone 31, detect eye areas according to this detected pupil.
The figure of the summary of applied eyeball phantom when Fig. 8 is the direction of visual lines that illustrates on the direction of visual lines calculating part 422 calculated level faces.The major part of the spherical eyeball 400 shown in this figure is covered by skin, and the zone of actual expression is from an E ' in face-image
1By pupil 401 point of arrival E '
2Circular arc.Point O '
EyeBe the center of eyeball 400, some O
EyeBe with an O '
EyeProject to the point on the perspective plane 500.And some I is that the center I ' with pupil 401 projects to the point on the perspective plane 500.In Fig. 8, the eyeball center O in establishing perspective plane 500
EyeAnd the distance between the pupil center location I is c
Eye, when the angle of hidden parts of establishing the horizontal direction of eyeball 400 is α, the center I ' of pupil 401 with respect to the direction of the optical axis direction (above-below direction of Fig. 8) of phtographic lens 21, be the direction of visual lines Φ on the horizontal plane
EyeProvide by following formula (3).
[numerical expression 3]
During the direction of visual lines of direction of visual lines calculating part 422 on calculating vertical plane, also adopt with the calculated level face on the identical eyeball phantom of the applied eyeball phantom of direction of visual lines.Therefore, with respect to the direction of visual lines of the vertical direction of the optical axis direction of phtographic lens 21
Provide by following formula (4).
[numerical expression 4]
Here, c '
EyeBe the eyeball center O in the perspective plane 500
EyeAnd the distance of the vertical direction between the pupil center location I, angle [alpha] ' be the angle of hidden parts of the vertical direction of eyeball 400.
Calculated at direction of visual lines calculating part 422 under the situation of direction of visual lines of eyes, the direction of visual lines weighted sum to the eyes that calculate to determine direction of visual lines thus.Weight when deciding summation according to facial direction here.For example, under the situation of face orientation right-hand (being viewed as left from camera head 1), right eye is not almost taken in the image, so, make the weight of left eye compare fully big with the weight of right eye.And, in the time of near the face orientation front, get direction of visual lines average of eyes.Like this, the direction of visual lines that direction of visual lines calculating part 422 calculates is inferred the result as sight line, be stored in the facial information storage part 132 of storage part 13.
In addition, the parameter when carrying out the direction of visual lines computing is not limited to above-mentioned parameter, and for example, the angle that also can establish the hidden parts of eyeball is different values at the central side of face with the outside.Comprise this in the interior facial direction calculating processing and the detailed content of direction of visual lines computing, for example open in above-mentioned patent documentation 2.
Infer portion 42 in sight line and carried out result that sight line among the step S33 infers circular treatment when calculating a plurality of direction of visual lines (step S34: be) at whole faces, camera head 1 enters step S35.Relative therewith, (step S34: not), camera head 1 finishes the sight line determination processing, enters step S4 described later when not calculating a plurality of direction of visual lines to infer result that sight line that portion 42 carried out step S33 at whole faces infers circular treatment in sight line.
Fig. 9 is the figure that is schematically illustrated near the situation of the explanation camera head 1 of the sight line determination processing time institute reference of step S35.Particularly, Fig. 9 observes the camera watch region (angle of visual field θ of horizontal direction that is present in camera head 1 from the top of camera head 1
X) in two personage A1, A2 watching the schematic diagram of the situation of same target T attentively.In Fig. 9, the position of establishing with the personage A1 of the distance of the phtographic lens 21 of camera head 1 is origin of coordinates O, establishes the axle parallel with the optical axis P of phtographic lens 21 and is the L axle, and on the other hand, the axle of establishing with the optical axis P quadrature of phtographic lens 21 is X-axis.And, in Fig. 9, establish the angle (Φ that is equivalent to formula (3) of direction of visual lines of the horizontal direction of personage A1, A2 respectively
Eye) be Φ 1, Φ 2.Under this meaning, in above-mentioned steps S332, calculated angle Φ 1, Φ 2, and it has been stored in the facial information storage part 132.
In step S35, distance calculation portion 43 is a starting point with phtographic lens 21, calculates apart from the distance (step S35) that calculates a plurality of personages of direction of visual lines respectively.Figure 10 illustrates the figure that calculates distance L 1, L2 conceptually.Distance calculation portion 43 is the distance L 1 by personage A1 is focused and calculates personage A1 for example, calculates the distance L 2 of personage A2 then according to this distance L 1.In addition, in Figure 10, the focal length of establishing phtographic lens 21 is F, and the height of establishing the face 202 of the height of face 201 of the personage A1 in the face-image and personage A2 is respectively H1 and H2, and facial 201,202 size all is similar to D.When using these definition, the distance L 1 from phtographic lens 21 to personage A1, A2, L2 are expressed as L1=DF/H1, L2=DF/H2 respectively.Therefore, distance L 2=(H2/H1) L1 can calculate by service range L1 in distance calculation portion 43.Distance L 1, L2 that distance calculation portion 43 is calculated are stored in the facial information storage part 132 of storage part 13.
Afterwards, intersecting detection unit 44 judges whether a plurality of personages' sight line intersects (step S36).At first, the detection unit 44 that intersects reads out in distance L 1, the L2 that is calculated by distance calculation portion 43 among the step S35 from facial information storage part 132, calculate the difference delta L of L coordinate of personage A1 and personage A2 and the difference delta X of X coordinate respectively by following formula (5) and (6).
ΔL＝L1-L2…(5)
ΔX＝L1tan(ρ1)+L2tan(ρ2)…(6)
Here, the ρ 1 (＜θ of formula (6)
X) be the direction of the personage A1 that observes from phtographic lens 21 and the optical axis P angulation of phtographic lens 21.And, the ρ 2 (＜θ of formula (6)
X) be the direction of the personage A2 that observes from phtographic lens 21 and the optical axis P angulation of phtographic lens 21.
Figure 11 is a schematic diagram of observing the situation identical with Fig. 9 from the side of camera head 1.In Figure 11, same with Fig. 9, the position of establishing with the big personage A1 of the distance of camera head 1 is origin of coordinates O, and the axle of establishing with the optical axis P quadrature of phtographic lens 21 is Y-axis.And in Figure 11, the angle of visual field of establishing the Y direction of camera head 1 is θ
YAnd then in Figure 11, the angle of direction of visual lines of establishing the horizontal direction of personage A1, A2 respectively (is equivalent to formula (4)
) be
Under this meaning, same with Φ 1 grade, in above-mentioned steps S332, calculated angle
And be stored in the facial information storage part 132.The detection unit 44 that intersects calculates the difference delta Y of the Y coordinate of personage A1 and personage A2 according to following formula (7).
ΔY＝L1tan(η1)+L2tan(η2)…(7)
Here, the η 1 (＜θ of formula (7)
Y) be the direction of the personage A1 that observes from phtographic lens 21 and the optical axis P angulation of phtographic lens 21.And, the η 2 (＜θ of formula (7)
Y) be the direction of the personage A2 that observes from phtographic lens 21 and the optical axis P angulation of phtographic lens 21.
Then, except the result of calculation of formula (5), (6), the detection unit 44 that intersects also uses angle Φ 1, the Φ 2 of direction of visual lines of the horizontal direction of the personage A1, the A2 that read from facial information storage part 132, according to following formula (8), (9), the intersecting point coordinate of the sight line of personage A1, A2 in the difference calculated level direction (Ls, Xs).
[numerical expression 5] Δ
And except the result of calculation of formula (5), (7), the detection unit 44 that intersects also uses the angle of direction of visual lines of the vertical direction of the personage A1, the A2 that read from facial information storage part 132
According to following formula (10), (11), calculate respectively the sight line of personage A1, A2 in the vertical direction intersecting point coordinate (Ls, Ys).
[numerical expression 6]
Then, whether the Ls of intersection detection unit 44 determine types (8) is consistent with the Ls of formula (10).In view of two Ls are presumed value, said here " unanimity " is meant that the difference of two Ls is the value in the prescribed limit.Under the Ls of formula (8) situation consistent with the Ls of formula (10), the detection unit 44 that intersects is judged to be the state that the sight line of the sight line that is in personage A1 and personage A2 is concentrated.
In addition, Ls is under the situation of ∞ in formula (8), (10), and it is parallel and crossing at unlimited distance that the detection unit 44 that intersects is judged to be direction of visual lines.And under the situation of Ls＜0, two people's sight line is intersected at the rear side of face in formula (8), (10), so the detection unit 44 that intersects is got rid of this situation from judge object.
Here, the situation that comprises the personage more than three people in view data is described.In this case, the detection unit 44 that intersects will be positioned at personage from phtographic lens 21 position farthest as the benchmark personage, use the difference of this benchmark personage and other personages' coordinate figure, intersect judgement.For example, comprise in the browse graph picture under n people's's (n is the natural number more than 3) personage's the situation, the detection unit 44 that intersects calculates the intersecting point coordinate of benchmark personage's direction of visual lines and other personages' direction of visual lines according to maximum (n-1).Then, intersect detection unit 44, judge respectively whether the Ls (formula (10)) of the Ls (formula (8)) of horizontal direction and vertical direction is consistent at a plurality of intersecting point coordinates that calculate.Under the situation of the consistent intersecting point coordinate (hereinafter referred to as " standard intersecting point coordinate ") of the Ls of Ls that to have a plurality of these result of determination be horizontal direction and vertical direction, the detection unit 44 that intersects judges whether a plurality of standard intersecting point coordinates are consistent each other.When carrying out this judgement, one of Rule Extraction should be as the standard intersecting point coordinate (benchmark intersecting point coordinate) of benchmark according to the rules to intersect detection unit 44, and calculates the difference of this benchmark intersecting point coordinate and in addition standard intersecting point coordinate respectively according to each component.The benchmark intersecting point coordinate of the value in the detection unit 44 that intersects is judged to be the standard intersecting point coordinate and the difference of each component is respectively prescribed limit is consistent.Under the situation that has the standard intersecting point coordinate consistent with the benchmark intersecting point coordinate more than the requirement ratio, the personage that the detection unit 44 that intersects is judged to be in the browse graph picture is in the state that sight line is concentrated.
Then, whether the processing after the sight line determination processing of the step S3 that more than illustrates is set at the automatic photography pattern and difference according to camera head 1.At first, illustrate that camera head 1 is not set at the situation of automatic photography pattern (step S4: not).In this case, when the sight line that is judged to be a plurality of personages in the browse graph picture in step S3 is concentrated (step S5: be), as intersecting the result of determination of detection unit 44, display part 5 shows that being used to report is that the OK of the situation of the sight line timing concentrating, be suitable for photographing shows (step S6).On the other hand, (step S5: not), camera head 1 is not exported OK via display part 5 and is shown, and directly enters step S7 described later under the situation that the sight line of a plurality of personages in not being judged to be the browse graph picture is concentrated.
Figure 12 schematically illustrates the routine figure of demonstration output that the OK in the display part 5 shows.Display image 51 shown in this figure illustrates the demonstration example under Fig. 9 and the situation shown in Figure 11.In display image 51, browse graph as the overlapping demonstration of upper left quarter " regularly OK " character string.Exported under the situation that OK shows showing on the display part 5, it is the situation of the sight line of personage A1, the A2 timing concentrating, be suitable for photographing that the cameraman can identify.In addition, in Figure 12, the record that the intersection point CP of the sight line of personage A1, A2 (with dashed lines demonstration) and two sight lines is only auxiliary.
Figure 13 is the figure that another demonstration output example (the 2nd example) of OK demonstration is shown.In the display image shown in this figure 52, overlapping demonstration represents that the concentrated personage of sight line of the personage's that sight line is concentrated square shape determines symbol 52a on the facial zone of personage A11, A12 in the browse graph picture, and overlapping demonstration sight line does not concentrate the personage to determine symbol 52b towards the sight line of the triangle of the direction different with other personages on the facial zone of the personage A13 in the browse graph picture.In the browse graph picture, exist under the personage's that the sight line more than the requirement ratio concentrates the situation, show that sight line concentrates the personage to determine that symbol 52a and sight line do not concentrate the personage to determine symbol 52b.In addition, in display part 5, also can show that the concentrated personage of sight line determines that the concentrated personage of symbol 52a and sight line determines symbol 52b with mutually different color.And, can at random set input via input part 9 by the user and be used on the browse graph picture overlapping demonstration sight line and concentrate the personage to determine that symbol 52a and sight line do not concentrate the personage to determine the condition (personage that concentrates of sight line is with respect to the ratio of subject integral body) of symbol 52b.
Figure 14 is the figure that the another demonstration output example (the 3rd example) of OK demonstration is shown.Display image 53 shown in this figure shows the result of the sight line judgement of having carried out 5 personage A21～A25.In display image 53, with browse the doubling of the image and show personage that the expression sight line concentrates has 4 people in 5 people message, and,, and browse the doubling of the image and show that sight line do not concentrate the personage to determine symbol 53a at the facial zone of the sight line personage A25 different with other 4 people.In the browse graph picture, exist under the personage's that the sight line more than the requirement ratio concentrates the situation, show this message and symbol.
Then, under the situation of having imported the photography index signal by the cameraman from input part 9 (step S7: be), camera head 1 is according to the control of control part 12, carry out the photography (step S8) of image, and utilize compressed and decompressed 7 and captured Image Data Compression is become given size, the Imagery Data Recording (step S9) in recording medium 100 after will compressing by recording medium interface 8.Here, camera head 1 can write and write down captured view data to storage part 13.In addition, also (step S7: not), camera head 1 returns step S3 under the situation of input photography index signal to have passed through the stipulated time in step S7.
Then, illustrate that camera head 1 is set at the situation of automatic photography pattern (step S4: be).In this case, when the result of the sight line determination processing (step S3) of image processing part 4 is that the sight line that is judged to be a plurality of personages in the browse graph picture is when concentrating (step S10: be), camera head 1 carries out photograph processing (step S8) automatically, utilize compressed and decompressed 7 captured Image Data Compression is become given size, the Imagery Data Recording (step S9) in recording medium 100 after will compressing by recording medium interface 8.On the other hand, (step S10: not), camera head 1 returns step S3 when the result of the sight line determination processing of image processing part 4 concentrates for the sight line that is not judged to be a plurality of personages in the browse graph picture.
Then, whether the processing of step S9 is set at burst mode and difference according to camera head 1.Be not set at camera head 1 that (step S11: not), image processing part 4 is read view data from recording medium interface 8, carries out the sight line determination processing (step S12) identical with above-mentioned steps S3 under the situation of burst mode.Here, carrying out the sight line determination processing once more is because from having time-delay to actual the photography after the sight line determination processing of step S3, might produce the result of determination different with the result of determination of step S3.
On the other hand, be set at camera head 1 under the situation of burst mode (step S11: be), if the photography of regulation photography number finishes (step S13: not), then return step S8 and photograph repeatedly as yet.If being set at the photography of burst mode (step S11: be) and regulation photography number, camera head 1 finishes (step S13: be), then image processing part 4 is read view data from recording medium interface 8, carries out the sight line determination processing (step S12) identical with above-mentioned steps S3.
In the result that image processing part 4 carries out sight line determination processing (step S12) is to be judged to be under the situation that the personage's who is comprised in the captured view data sight line concentrates (step S14: be), camera head 1 is concentrated image with this view data as sight line, records (step S15) in the recording medium 100 via recording medium interface 8.Here said " sight line is concentrated image " is to be endowed the view data of representing the information of the situation that sight line is concentrated, and for example, setting up expression at raw image data is the sign that sight line is concentrated image.In addition, be to be judged to be not that (step S14: not), camera head 1 returns step S3 under the situation that the sight line of the personage in the captured view data concentrates in the result that image processing part 4 carries out the sight line determination processing.
Then, be set at camera head 1 under the situation of cutting pattern (step S16: be), cutting portion 45 carries out the cutting (step S17) that sight line is concentrated image.Then, display part 5 shows by the result (step S18) after 45 cuttings of cutting portion.Figure 15 is the figure that the demonstration example of the cutting result in the display part 5 is shown.Cutting shown in this figure display image 54 as a result makes the with dashed lines demonstration only extract the personage A11 that sight line is concentrated, the zone of A12, as clipping region 54a.In addition, in step S18, also can replace display part 5 to show the cutting situation of display image 54 as a result, and only show clipping region 54a.And, also can be used to inquire the message that whether writes down the cutting image in the cutting appropriate location output of display image 54 as a result, supervise the cameraman to carry out importing from the selection of input part 9.
Under the situation of the operation index signal of having imported record cutting image from input part 9 (step S19: be), in camera head 1, utilize compressed and decompressed 7 the images cut data compression is become given size, the cutting Imagery Data Recording (step S20) in recording medium 100 after will compressing by recording medium interface 8.On the other hand, (step S19: not), camera head 1 finishes a series of processing under the situation of the operation index signal of not importing record cutting image.
Be not set at camera head 1 that (step S16: not), camera head 1 finishes a series of processing under the situation of cutting pattern.
Then, illustrate that camera head 1 is set at photograph mode (step S1: photograph mode) and not be set at sight line and judge the situation of photograph mode (step S2: not).In this case, camera head 1 carries out common photograph processing (step S21).
Then, usually whether the processing of photograph processing is set at photography according to camera head 1 and judges the cutting pattern and difference with sight line.Be set at photography at camera head 1 and judge that with sight line image processing part 4 carries out the sight line determination processing (step S23) identical with step S12 at captured view data under the situation of cutting pattern (step S22: be).Relative therewith, be not set at photography at camera head 1 and judge that with sight line (step S22: not), camera head 1 only carries out common photograph processing under the situation of cutting pattern.
Below, illustrate that camera head 1 is set at the cutting pattern is judged in photography with sight line situation.In this case, the result who carries out sight line determination processing (step S23) at image processing part 4 is that the personage's that is judged to be in the view data to be comprised sight line is when concentrating (step S24: be), camera head 1 is concentrated image with this view data as sight line, records (step S25) in the recording medium 100 via recording medium interface 8.Then, cutting portion 45 carries out the cutting (step S17) that sight line is concentrated image.Then the processing of step S18～S20 of this step S17 as mentioned above.
On the other hand, the result who carries out the sight line determination processing at image processing part 4 is that (step S24: not), camera head 1 finishes a series of processing when concentrating for the personage's that is not judged to be in the view data to be comprised sight line.
Then, illustrate that camera head 1 is set at the situation of reproduction mode (step S1: reproduction mode).At first, display part 5 reproduces and show the view data of being selected by the cameraman (step S26) from the view data that recording medium 100 is write down.
Be set to reproduction at camera head 1 and judge with sight line under the situation of cutting pattern (step S27: be) that 4 pairs of image processing parts reproduce the view data that shows and carry out sight line determination processing (step S23).Then the processing from step S24 of this step S23 is set at photograph mode with camera head 1 and photographs and judge that with sight line the situation of cutting pattern is identical.
On the other hand, be not set at reproduction at camera head 1 and judge (step S27: deny) under the situation of cutting pattern, camera head 1 end process with sight line.
An embodiment of the invention according to above explanation, infer the sight line of a plurality of personages in the picture signal, calculate respectively apart from the distance of inferring out a plurality of personages of sight line, use these sight lines to infer result and distance calculation result, whether the sight line of judging a plurality of personages intersects, so, can extract the scene that most of personages watch the same object thing attentively in comprising the image of a plurality of personages as subject reliably.
So far, illustrated to be used to implement optimal way of the present invention, still, the invention is not restricted to an above-mentioned execution mode.For example, the intersection point of a plurality of personages' sight line is entered under the situation in the browse graph picture that shows on display part 5, also can on display part 5, show this intersection point.Figure 16 illustrates the cameraman to enter the figure of the demonstration example the display part 5 under the situation of area of visual field of camera head 1 from state shown in Figure 9 (with reference to the display image 51 of Figure 12) to L axle positive direction dollying device 1, terminal objective T.Display image 55 shown in Figure 16 shows the intersection point CP of the sight line of the sight line of personage A1 and personage A2 by the star mark.Therefore, the cameraman can easily identify a plurality of personages' sight line position that intersects and the object that is present in this crossover location place.
Like this, the present invention comprises here the various execution modes of not record etc., can implement various design alterations etc. in the scope that does not break away from by the determined technological thought of claim scope.
Claims (7)
1. an image processing apparatus is characterized in that, this image processing apparatus has:
Facial test section, it detects a plurality of personages' that comprised in the described picture signal face according to the image model in the picture signal;
Sight line is inferred portion, and it infers the sight line that is detected facial described a plurality of personages by described facial test section;
Distance calculation portion, it calculates the distance of inferring out described a plurality of personages of sight line apart from by the described sight line portion of inferring respectively; And the intersection detection unit, it uses described sight line to infer the result of calculation of inferring result and described distance calculation portion of portion, judges whether described a plurality of personages' sight line intersects.
2. image processing apparatus according to claim 1 is characterized in that,
This image processing apparatus has display part, and this display part shows the view data corresponding with described picture signal,
The result of determination of described display part described intersection detection unit of overlapping demonstration on described view data.
3. image processing apparatus according to claim 2 is characterized in that,
The overlapping demonstration on described view data of described display part is used for determining the personage's of intersecting in the mutual sight line of described a plurality of personages information.
4. image processing apparatus according to claim 1 is characterized in that,
This image processing apparatus has:
Image pickup part, it is made a video recording to the subject in the area of visual field of regulation, and generates picture signal;
Input part, its input are used to indicate the photography index signal that the captured subject of described image pickup part is photographed; And
Control part, it is according to the control of carrying out from the described photography index signal of described input part input photographing to by the captured subject of described image pickup part.
5. image processing apparatus according to claim 4 is characterized in that,
Described control part carries out the control that exposes and/or focus by the detected face of described facial test section.
6. according to claim 4 or 5 described image processing apparatus, it is characterized in that,
Described control part carries out the automatic control that the captured subject of described image pickup part is photographed according to the result of determination of described intersection detection unit.
7. an image processing method is characterized in that, this image processing method is the image processing method that picture signal is carried out the image processing apparatus of predetermined processing, may further comprise the steps:
Face detection step in this step, according to the image model in the described picture signal, detects a plurality of personages' that comprised in the described picture signal face;
Sight line is inferred step, in this step, infers the sight line that detects facial described a plurality of personages in described face detection step, is stored to the facial information storage part that is used for storing facial information;
The distance calculation step in this step, is calculated the distance of inferring the described a plurality of personages that infer out sight line in the step apart from described sight line respectively, is stored in the described facial information storage part; And intersection determination step, in this step, read described sight line and infer the result of calculation of inferring result and described distance calculation step of step from described facial information storage part, that uses that this reads infers result and result of calculation, judges whether described a plurality of personages' sight line intersects.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2008079122 | 2008-03-25 | ||
| JP2008-079122 | 2008-03-25 | ||
| JP2008079122A JP5042896B2 (en) | 2008-03-25 | 2008-03-25 | Image processing apparatus and image processing program |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101547307A CN101547307A (en) | 2009-09-30 |
| CN101547307B true CN101547307B (en) | 2011-07-06 |
Family
ID=41194152
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2009101268897A Active CN101547307B (en) | 2008-03-25 | 2009-03-24 | Image processing device and program |
Country Status (2)
| Country | Link |
|---|---|
| JP (1) | JP5042896B2 (en) |
| CN (1) | CN101547307B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5794797B2 (en) * | 2011-03-18 | 2015-10-14 | オリンパス株式会社 | Image display device and imaging device |
| JP5719223B2 (en) * | 2011-04-25 | 2015-05-13 | オリンパスイメージング株式会社 | Image recording apparatus, recording method, and program |
| CN105005777B (en) * | 2015-07-30 | 2021-02-02 | 科大讯飞股份有限公司 | Audio and video recommendation method and system based on human face |
| CN105894031A (en) * | 2016-03-31 | 2016-08-24 | 青岛海信移动通信技术股份有限公司 | Photo selection method and photo selection device |
| JP6631607B2 (en) * | 2017-09-20 | 2020-01-15 | カシオ計算機株式会社 | Image processing apparatus, image processing method, and program |
| JP6787299B2 (en) * | 2017-11-15 | 2020-11-18 | オムロン株式会社 | Alarm control device, alarm control method, and program for alarm control |
| CN110045834A (en) * | 2019-05-21 | 2019-07-23 | 广东工业大学 | Detection method, device, system, equipment and storage medium for sight locking |
| CN112989865B (en) * | 2019-12-02 | 2023-05-30 | 山东浪潮科学研究院有限公司 | Crowd attention focus judging method based on head gesture judgment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001051338A (en) * | 1999-08-12 | 2001-02-23 | Canon Inc | Camera |
| CN1522052A (en) * | 2003-01-17 | 2004-08-18 | ŷķ����ʽ���� | Image pickup device |
| JP2007006427A (en) * | 2005-05-27 | 2007-01-11 | Hitachi Ltd | Video monitor |
| JP2007096440A (en) * | 2005-09-27 | 2007-04-12 | Casio Comput Co Ltd | Imaging apparatus |
| WO2008007781A1 (en) * | 2006-07-14 | 2008-01-17 | Panasonic Corporation | Visual axis direction detection device and visual line direction detection method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006201531A (en) * | 2005-01-21 | 2006-08-03 | Konica Minolta Holdings Inc | Imaging device |
| JP2006319610A (en) * | 2005-05-12 | 2006-11-24 | Matsushita Electric Ind Co Ltd | Camera |
| JP2007306463A (en) * | 2006-05-15 | 2007-11-22 | Fujifilm Corp | Trimming supporting method, apparatus, and program |
| JP4874735B2 (en) * | 2006-08-04 | 2012-02-15 | 綜合警備保障株式会社 | Security device, monitoring system, abnormality determination method and abnormality determination program |
-
2008
- 2008-03-25 JP JP2008079122A patent/JP5042896B2/en not_active Expired - Fee Related
-
2009
- 2009-03-24 CN CN2009101268897A patent/CN101547307B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001051338A (en) * | 1999-08-12 | 2001-02-23 | Canon Inc | Camera |
| CN1522052A (en) * | 2003-01-17 | 2004-08-18 | ŷķ����ʽ���� | Image pickup device |
| JP2007006427A (en) * | 2005-05-27 | 2007-01-11 | Hitachi Ltd | Video monitor |
| JP2007096440A (en) * | 2005-09-27 | 2007-04-12 | Casio Comput Co Ltd | Imaging apparatus |
| WO2008007781A1 (en) * | 2006-07-14 | 2008-01-17 | Panasonic Corporation | Visual axis direction detection device and visual line direction detection method |
Also Published As
| Publication number | Publication date |
|---|---|
| JP5042896B2 (en) | 2012-10-03 |
| CN101547307A (en) | 2009-09-30 |
| JP2009239347A (en) | 2009-10-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101547307B (en) | Image processing device and program | |
| KR101539043B1 (en) | Image photography apparatus and method for proposing composition based person | |
| CN105740757B (en) | The control method of zooming control device, picture pick-up device and zooming control device | |
| US7038715B1 (en) | Digital still camera with high-quality portrait mode | |
| CN100556078C (en) | Camera head, image processing apparatus and image processing method | |
| CN105744149B (en) | The control method of zooming control device, picture pick-up device and zooming control device | |
| CN101783019B (en) | Subject tracking apparatus and control method therefor, image capturing apparatus, and display apparatus | |
| US20160335495A1 (en) | Apparatus and method for acquiring image for iris recognition using distance of facial feature | |
| CN110536069A (en) | The control method and recording medium of zooming control device, zooming control device | |
| US8290276B2 (en) | Image display device, image display method and storage medium storing image display program | |
| US8400532B2 (en) | Digital image capturing device providing photographing composition and method thereof | |
| CN104253984A (en) | Information processor and information processing method | |
| CN105407268B (en) | Image correction apparatus, method for correcting image and recording medium | |
| CN108897423A (en) | A kind of VR glasses and its online testing anti-cheating method | |
| JP4449354B2 (en) | Image photographing apparatus and program | |
| CN101262561A (en) | Imaging apparatus and control method thereof | |
| KR20170074742A (en) | Image processing device, image processing method amd program | |
| CN103379279A (en) | Subject area detection apparatus and method therefor, as well as image pickup apparatus and display apparatus | |
| JP4896311B2 (en) | Fundus image acquisition and display device | |
| CN101561872B (en) | Image processing apparatus | |
| JP2007049631A (en) | Imaging apparatus | |
| JP5109864B2 (en) | Electronic still camera | |
| JP2010068190A (en) | Digital camera, image processing apparatus and digital camera system | |
| JP4063583B2 (en) | Digital camera | |
| JP2013118574A (en) | Imaging apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160222 Address after: Tokyo, Japan, Japan Patentee after: Olympus Corporation Address before: Tokyo, Japan Patentee before: Olympus Imaging Corp. Patentee before: Olympus Corporation |