JP6390210B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

In recent years, a so-called “virtual mannequin” that projects an image on a projection screen molded in a human shape has been proposed (for example, see Patent Document 1). The virtual mannequin provides a projected image that has the presence of a human being standing there, and can be used to create innovative and effective exhibitions at exhibitions and the like.
In order to enrich the facial expression of such a virtual mannequin, there is a technology that expresses movement by transforming the main parts (for example, eyes and mouth) that make up the face in images such as photographs, illustrations, and comics. Are known. Specifically, based on the gaze point of the human being who is the subject, a method of operating the eyeball in the face model represented by the human computer graphics (for example, see Patent Document 2), There is a method of changing the shape of the mouth for each vowel or consonant to perform lip synchronization (see, for example, Patent Document 3).

JP 2011-150221 A Japanese Patent Laid-Open No. 06-282627 JP 2003-58908 A

  By the way, when operating the main part of the face, the image to be processed needs to include the face. That is, when an image that does not include a face, or an image with low face detection or a low detection accuracy of the main part of the face is a processing target, there is a problem that the main part of the face cannot be deformed to express a motion. is there.

  The present invention has been made in view of such problems, and an object of the present invention is to provide an image processing apparatus and an image processing method capable of setting a main part of a face even for an image that does not include a face. And providing a program.

In order to solve the above problems, an image processing apparatus according to the present invention provides:
An acquisition means for acquiring an image;
A region detecting unit for detecting a region corresponding to the shape of the main part of the face from a region not including the face image in the image acquired by the acquiring unit;
A first setting means for setting a region detected by the region detecting means as images processed,
It is characterized by having.

  According to the present invention, the main part of a face can be set even for an image that does not include a face.

It is a block diagram which shows schematic structure of the image processing apparatus of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to an operation process performed by the image processing apparatus in FIG. 1. 3 is a flowchart showing an example of an operation related to a main part setting process in the operation process of FIG. 2. It is a figure which shows typically the image which concerns on the operation | movement process of FIG. It is a figure which shows typically the image which concerns on the operation | movement process of FIG. It is a figure which shows typically the image which concerns on the operation | movement process of FIG. It is a figure which shows typically the image which concerns on the operation | movement process of FIG.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus 100 according to an embodiment to which the present invention is applied.

The image processing apparatus 100 includes, for example, a computer such as a personal computer or a workstation. As shown in FIG. 1, the central processing unit 1, a memory 2, a storage unit 3, an operation input unit 4, and an operation are performed. A processing unit 5, a display unit 6, and a display control unit 7 are provided.
The central control unit 1, the memory 2, the storage unit 3, the operation processing unit 5, and the display control unit 7 are connected via a bus line 8.

The central control unit 1 controls each unit of the image processing apparatus 100.
Specifically, the central control unit 1 includes a CPU (Central Processing Unit; not illustrated), a RAM (Random Access Memory), and a ROM (Read Only Memory) that control each unit of the image processing apparatus 100. Various control operations are performed in accordance with various processing programs (not shown).

  The memory 2 is configured by, for example, a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores data processed by each unit of the image processing apparatus 100 in addition to the central control unit 1.

  The storage unit 3 includes, for example, a nonvolatile memory (flash memory), a hard disk drive, and the like, and stores various programs and data (not shown) necessary for the operation of the central control unit 1.

The storage unit 3 stores image data 3a.
The image data 3a is, for example, data of a two-dimensional image I (see FIG. 4A) such as a photographic image, a cartoon, and an illustration. The image data 3a may be image data of an image including a face or image data of an image not including a face.

The storage unit 3 stores reference operation data 3b.
The reference motion data 3b includes information indicating movement that serves as a reference when expressing the movement of each major part (for example, eyes, mouth, etc.) of the face. Specifically, the reference motion data 3b is defined for each main part and includes information indicating movements of a plurality of control points in a predetermined space. For example, the position coordinates (x , y) and information indicating deformation vectors are arranged along the time axis.
That is, for example, in the eye reference motion data 3b, a plurality of control points corresponding to the upper eyelid and the lower eyelid are set, and deformation vectors of these control points are defined. For example, in the mouth reference motion data 3b, a plurality of control points corresponding to the upper lip, the lower lip, and the left and right mouth corners are set, and a deformation vector of these control points is defined.

The storage unit 3 stores face main part image data 3c.
The face main part image data 3c is defined for each main part of the face, such as each of the left and right eyes, nose, mouth, eyebrows, and face contour, and the corresponding main part image P (see FIG. 7B; see below). It is the image data concerning.
The main part image P is prepared in various shapes and colors such as an image extracted from the face of a photographic image and an image schematically showing the main part. In addition, a plurality of control points for operating the main part image are set at predetermined positions of the main part image P in association with a plurality of control points of the reference action data 3b. Yes.

  The operation input unit 4 includes, for example, a data input key for inputting numerical values, characters, and the like, a keyboard, a mouse, and the like configured by up / down / left / right movement keys and various function keys for performing data selection, feed operation, and the like. The operation unit (not shown) is provided, and a predetermined operation signal is output to the central control unit 1 in accordance with the operation of these operation units.

The motion processing unit 5 includes an image acquisition unit 5a, a face detection unit 5b, a face main part detection unit 5c, an area detection unit 5d, a first main part setting unit 5e, an area specifying unit 5f, and a second main part. A unit setting unit 5g, a motion generation unit 5h, and a motion control unit 5i.
In addition, although each part of the operation | movement process part 5 is comprised from the predetermined logic circuit, for example, the said structure is an example and is not restricted to this.

The image acquisition unit 5a acquires the image data 3a.
That is, the image acquisition unit (acquisition unit) 5a acquires the image data 3a of the two-dimensional image I (see FIG. 4A and the like) that is the processing target of the operation process. Specifically, the image acquisition unit 5a is, for example, a user-desired image specified based on a predetermined operation of the operation input unit 4 by the user in a predetermined number of image data 3a stored in the storage unit 3, for example. Data 3a is acquired as a processing target of the operation process.
The image acquisition unit 5a may acquire image data from an external device (not shown) connected via a communication control unit (not shown), or an image generated by being captured by an imaging unit (not shown). Data may be acquired.

The face detection unit 5b detects a face area from the processing target image I.
That is, the face detection unit 5b detects a face area including a face from the image data of the processing target image I acquired by the image acquisition unit 5a. Specifically, the face detection unit 5b first acquires a difference from the image of the background portion in the image I acquired by the image acquisition unit 5a, and recognizes the object region. Then, the face detection unit 5b is a region having characteristics similar to a face in the object region, for example, each of three circular or polygonal shape portions B,. A region having a positional relationship in which the center is arranged at the position of the vertex of the triangle is searched. Then, the face detection unit 5b performs a predetermined calculation on the searched area based on the size of each part, the distance between the centers, and the positional relationship to calculate an evaluation value, and the calculated evaluation value is a predetermined threshold value. A region having three circular or polygonal shape portions B,... That is higher and has the highest evaluation value is detected as a face region of the object.
The face area detection process described above is an example and is not limited to this, and can be arbitrarily changed as appropriate. The face detection unit 5b uses, for example, an AAM (Active Appearance Model) or the like. A detection process may be performed.

The face main part detection unit 5c detects the main part constituting the face from the image I.
That is, the face main part detection unit (detection unit) 5b detects, for example, main parts such as left and right eyes, nose, mouth, eyebrows, and face contour from the image I of the image data acquired by the image acquisition unit 5a. To do. Specifically, the face main part detection unit 5c uses, for example, the result of the face area detection processing by the face detection unit 5b to extract the main part of the face from three circular or polygonal shape parts B,. To detect. For example, when an evaluation value equal to or greater than a predetermined threshold value is calculated in the face area detection process described above, the face main part detection unit 5c determines three circular or polygonal shape portions B,. It detects as both eyes and a mouth (refer FIG.4 (b)).
Note that the above-described method for detecting the main part of the face is an example, and is not limited to this. For example, processing using AAM, edge extraction processing, anisotropic diffusion processing, template matching, and the like can be arbitrarily arbitrarily performed. It can be changed.

The area detection unit 5d detects the main part equivalent area A1 in the image I.
That is, the region detection unit (region detection unit) 5d detects a region (main part equivalent region A1) corresponding to the shape of the main part of the face in the image I of the image data acquired by the image acquisition unit 5a. Specifically, the region detection unit 5d performs, for example, predetermined image processing (for example, Hough conversion processing) on the image data of the image I to be processed to form a circle or a polygon in the image I. The area is detected as the main part equivalent area A1. At this time, the area detection unit 5d may detect the main part equivalent area A1 using the detection result of the circular or polygonal shape part B by the face detection unit 5b.
In addition, the area detection unit 5d includes the image I in the image I acquired by the image acquisition unit 5a when no face is detected by the face detection unit 5b or when no main part is detected by the face main part detection unit 5c. The main part equivalent area A1 may be detected.

  If no main part equivalent area A1 having a circular or polygonal shape is detected from the image I, the area detecting unit 5d performs predetermined image processing (for image data of the image I to be processed). For example, edge detection processing or the like is performed to detect the widest area (for example, a substantially circular area) in which no edge exists in the image I, and the detected area is used as a main part (for example, eyes or mouth). ) May be detected as the set face area C (see FIG. 7A). At this time, the area detection unit 5d detects the position where the main part is set in consideration of the size of the detected area (face area C) and the positional relationship between the main parts such as the left and right eyes and the mouth. May be.

  In addition, although the area | region which makes circular or a polygon is detected as main part equivalent area | region A1, it is an example and is not restricted to this, For example, the area | region (for example, it is not completely enclosed by the line) The region detection unit 5d may estimate and detect a go region in which the end portions are virtually connected to each other even in the case of a “C” or “U” -shaped region.

The first main part setting unit (first setting means) 5e sets the area detected by the area detection unit 5d as an operation processing target (image processing target) of any main part.
That is, the first main part setting unit 5e superimposes an image (first main part image P1) representing a main part (for example, eyes, mouth, etc.) on the main part equivalent area A1 detected by the area detection part 5d. Set to.

For example, when two main part equivalent areas A1 and A1 having a circular or polygonal shape are detected from the image I by the area detection unit 5d, the first main part setting unit 5e detects the two main parts detected. The corresponding areas A1 and A1 are set as the operation processing targets of both the left and right eyes (main part) (see FIG. 5A). Specifically, the first main part setting unit 5e reads out and acquires the face main part image data 3c corresponding to the left and right eyes from the storage unit 3, and each of the first main part images P1 representing the left and right eyes has two. Of the first main part image P1 so that it falls within one main part equivalent area A1, A1 and the center of each first main part image P1 overlaps the center of each of the two main part equivalent areas A1, A1. The size and position are set (see FIG. 5B).
For example, when one region corresponding to a main portion A1 having a circular shape or a polygon is detected from the image I by the region detection unit 5d, the first main portion setting unit 5e detects the one main portion detected. The equivalent area A1 is set as an operation processing target for the mouth (main part) (see FIG. 6A). Specifically, the first main part setting unit 5e reads out and acquires the face main part image data 3c corresponding to the mouth from the storage unit 3, and the first main part image P1 representing the mouth is one main part equivalent region. The size and position of the first main part image P1 are set so that they fall within A1 and the center of the first main part image P1 and the center of one main part equivalent region A1 overlap (FIG. 6B). reference).

  Note that the orientations and arrangements of the two main part equivalent regions A1 and A1 described above are merely examples, and are not limited thereto, and can be arbitrarily changed as appropriate. For example, when a face in landscape orientation is expressed in the image I, the motion processing target of both eyes may be set using two main part equivalent areas A1 and A1 arranged vertically. In addition, when one main part equivalent area A1 is detected by the area detection unit 5d, the first main part setting part 5e sets the main part equivalent area A1 as the operation processing target of either the left or right eye. May be.

The area specifying unit 5f specifies the main part corresponding area A2 in the image I.
That is, the region specifying unit (specifying unit) 5f uses a position other than the main part set by the first main part setting unit 5e on the basis of the position of the main part equivalent area A1 detected by the area detection unit 5d in the image I. An area corresponding to another main part (main part corresponding area A2) is specified.
For example, the area specifying unit 5f corresponds to one main part corresponding to the mouth as the other main part on the basis of the positions of the two main part equivalent areas A1 and A1 detected by the area detecting unit 5d in the image I. The area A2 is specified (see FIG. 5A). Specifically, since the left and right eyes are set as the main parts in the two main part equivalent areas A1 and A1 in the image I by the first main part setting part 5e, the area specifying part 5f A position between two main part equivalent areas A1 and A1 in the image I and at a predetermined distance from the lower side with reference to the positional relationship and size of the part equivalent areas A1 and A1 in the image I One main part corresponding area A2 corresponding to the mouth is specified.
Further, for example, the area specifying unit 5f uses the two main corresponding to the left and right eyes as the other main parts with reference to the position of one main part equivalent area A1 detected by the area detecting unit 5d in the image I. The part corresponding areas A2 and A2 are specified (see FIG. 6A). Specifically, since the mouth is set as a main part in one main part equivalent area A1 in the image I by the first main part setting part 5e, the area specifying part 5f uses this one main part equivalent area A1. The two main part corresponding areas A2 corresponding to the left and right eyes at a position a predetermined distance above the one main part corresponding area A1 in the image I on the basis of the positional relationship, size, etc. in the image I , A2 is specified.

  Note that the orientation and arrangement of the two main part corresponding regions A2 and A2 described above are merely examples, and are not limited thereto, and can be arbitrarily changed as appropriate. For example, when a landscape face is represented in the image I, the two main part corresponding areas A2 and A2 may be specified so as to be arranged vertically.

The second main part setting unit (second setting means) 5g sets the main part corresponding area A2 specified by the area specifying part 5f as an operation processing target of another main part. That is, the second main part setting unit 5g performs setting so that an image representing the other main part (second main part image P2) is superimposed on the main part corresponding area A2 specified by the area specifying unit 5f.
For example, when one main part corresponding area A2 is specified in the image I by the area specifying part 5f, the second main part setting part 5g uses the specified one main part corresponding area A2 as a mouth (other It is set as the operation processing target of the main part. Specifically, the second main part setting unit 5g reads and acquires the face main part image data 3c corresponding to the mouth from the storage unit 3, and the second main part image P2 representing the mouth is one main part corresponding region. The size and position of the second main part image P2 are set so that they fall within A2 and the center of the second main part image P2 and the center of one main part corresponding region A2 overlap (FIG. 5B). reference).
Further, for example, when the two main part corresponding areas A2 and A2 are specified in the image I by the area specifying unit 5f, the second main part setting unit 5g includes the two specified main part corresponding areas A2, A2 is set as an operation processing target for both the left and right eyes (other main parts). Specifically, the second main part setting unit 5g reads out and acquires the face main part image data 3c corresponding to the left and right eyes from the storage unit 3, and each second main part image P2 representing both the left and right eyes is stored in two. Of the second main part image P2 so that it falls within one main part corresponding area A2, A2 and the center of each second main part image P2 and the center of each of the two main part corresponding areas A2, A2 overlap. The size and position are set (see FIG. 6B).

The contents set by the first main part setting unit 5e and the second main part setting unit 5g may be output to predetermined storage means (for example, the memory 2) and temporarily stored.
Furthermore, although eyes and mouths are illustrated as the main parts, this is an example, and the present invention is not limited to this. For example, other main parts such as a nose, eyebrows, and a face outline may be used.

The motion generation unit 5h generates motion data for operating the main part (for example, both the left and right eyes and the mouth) in the image I.
Specifically, the motion generation unit 5h is, for example, the main part related to the first main part image P1 set by the first main part setting unit 5e or the second main part set by the second main part setting unit 5g. The reference motion data 3b corresponding to the other main part related to the image P2 is read from the storage unit 3 and acquired. Then, the motion generation unit 5h includes the first main part image P1 and the second main part image P2 in the image I to be processed set by the first main part setting unit 5e and the second main part setting unit 5g. The information indicating the size and position and the information indicating the movements of a plurality of control points in the predetermined space defined in the reference motion data 3b are associated with each other in the first main image P1 and the second main image P2. Generate motion data.
The motion data generated by the motion generation unit 5h may be output to a predetermined storage unit (for example, the memory 2) and temporarily stored.

The operation control unit 5 i operates the main part in the image I.
That is, the motion control unit (motion control means) 5i represents an image (first main part image P1) representing the main part set by the first main part setting part 5e in the image I acquired by the image acquisition part 5a. And an image representing the other main part set by the second main part setting unit 5g (second main part image P2) is operated. Specifically, the motion control unit 5i generates the motion data of the first main part image P1 representing the main part to be processed generated by the motion generation unit 5h and the second main part image P2 representing the other main part. By acquiring the operation data and displacing the plurality of control points based on the information indicating the movement of the plurality of control points defined in the acquired operation data, the first main image P1 and the second main image An operation process for operating the partial image P2 is performed.
Note that the processing for superimposing the first main part image P1 and the second main part image P2 in the image I is performed using, for example, alpha blending or the like. The detailed explanation is omitted.

  The display unit 6 includes a display such as an LCD (Liquid Crystal Display) and a CRT (Cathode Ray Tube), and displays various types of information on the display screen under the control of the display control unit 7.

The display control unit 7 performs control to generate display data and display it on the display screen of the display unit 6.
Specifically, the display control unit 7 includes a video card (not shown) including, for example, a GPU (Graphics Processing Unit), a VRAM (Video Random Access Memory), and the like. Then, in accordance with the display instruction from the central control unit 1, the display control unit 7 generates display data for various screens for operating the main part in the operation process by drawing processing using a video card, and outputs it to the display unit 6. To do. Thereby, the display part 6 displays the content deform | transformed so that the principal part image P showing a principal part (for example, eyes, a mouth, etc.) by operation | movement processing might be operated, for example.

<Operation processing>
Next, the operation process will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the operation process.

As shown in FIG. 2, first, the image acquisition unit 5 a of the motion processing unit 5 performs, for example, a predetermined operation of the operation input unit 4 by the user among a predetermined number of image data 3 a stored in the storage unit 3. The user-desired image data 3a designated based on this is acquired (step S1).
Next, the motion processing unit 5 performs a main part setting process (see FIG. 3) for setting the main part of the face in the image I of the image data acquired by the image acquisition unit 5a (step S2).

<Main part setting process>
The main part setting process will be described in detail below with reference to FIG.
FIG. 3 is a flowchart showing an example of an operation related to the main part setting process.

  As shown in FIG. 3, first, the face detection unit 5b performs a process of detecting a face region including a face from the image data of the image I to be processed acquired by the image acquisition unit 5a (step S11). Specifically, the face detection unit 5b, for example, in the image I, as a part corresponding to the left and right eyes and mouth, each of the three circular or polygonal shape parts B,. A region having a positional relationship that is arranged at a position is searched, and three circular or polygonal shape portions B with the evaluation value of the searched region being higher than a predetermined threshold value and the highest evaluation value are obtained. A region having the same is detected as a face region of the object.

Subsequently, the face detection unit 5b determines whether or not a face area has been detected (step S12).
If it is determined that a face area has been detected (step S12; YES), the face main part detection unit 5c detects the main part of the face (for example, left and right eyes and mouth) from the image I to be processed. Is performed (step S13). Specifically, the face main part detection unit 5c, for example, if an evaluation value equal to or greater than a predetermined threshold is calculated in the face area detection processing by the face detection unit 5b, the three circular or polygonal shape parts B ,... Are detected as the left and right eyes and mouth of the object (see FIG. 4B).

Next, the face main part detection unit 5c determines whether or not the main part of the face has been detected (step S14).
Here, if it is determined that the main part of the face has been detected (step S14; YES), the motion processing unit 5 can use the detected main part of the face as an operation target. A main part extracted image Q is generated by extracting a main part (for example, both left and right eyes and a mouth) using processing (for example, edge extraction process) (step S15).
Note that the above-described edge extraction processing is a known technique, and thus detailed description thereof is omitted here.

On the other hand, if it is determined in step S14 that the main part of the face has not been detected (step S14; NO), the area detection unit 5d performs a process of detecting the main part equivalent area A1 in the image I ( Step S16). Specifically, the region detection unit 5d performs, for example, predetermined image processing (for example, Hough conversion processing) on the image data of the image I to be processed to form a circle or a polygon in the image I. The area is detected as the main part equivalent area A1.
Even when it is determined in step S12 that the face area is not detected (step S12; NO), the area detection unit 5d moves the process to step S14 and detects the main part equivalent area A1. Perform the process.

  Subsequently, the operation processing unit 5 branches the process according to the number of detected main part equivalent regions A1 (step S17). That is, when the number of detected main part equivalent areas A1 is two (step S17; two), the operation processing unit 5 shifts the process to step S181, and when the number of detected main part equivalent areas A1 is one (step S17). S17; one), the process proceeds to step S191, and when the number of detected main part equivalent area A1 is 0 (zero) (step S17; 0), the process proceeds to step S201.

<Number of detections; two>
When two main part equivalent areas A1 and A1 are detected by the area detection unit 5d (step S17; two), the first main part setting part 5e moves the two main part equivalent areas A1 and A1 left and right. Is set as an operation processing target for both eyes (main part) (step S181; see FIG. 5A). Specifically, the first main part setting unit 5e reads out and acquires the face main part image data 3c corresponding to the left and right eyes from the storage unit 3, and each of the first main part images P1 representing the left and right eyes has two. The size and position of the first main part image P1 are set so as to be arranged in the two main part equivalent areas A1 and A1 (see FIG. 5B).

Next, the area specifying unit 5f specifies one main part corresponding area A2 corresponding to the mouth in the image I (step S182; see FIG. 5A). Specifically, the area specifying unit 5f uses one main part corresponding area corresponding to the mouth in the image I based on the positional relationship, size, and the like in the image I of the two main part equivalent areas A1 and A1. A2 is specified.
Subsequently, the second main part setting unit 5g sets the specified one main part corresponding area A2 as an operation processing target of the mouth (other main part) (step S183). Specifically, the second main part setting unit 5g reads and acquires the face main part image data 3c corresponding to the mouth from the storage unit 3, and the second main part image P2 representing the mouth is one main part corresponding region. The size and position of the second main part image P2 are set so as to be arranged in A2 (see FIG. 5B).

  When three or more main part equivalent areas A1,... Are detected in step S16, the first main part setting part 5e determines the detected three or more main part equivalent areas A1,. In consideration of the size and the positional relationship in the image I, any two main part equivalent areas A1, A1 may be automatically designated from these main part equivalent areas A1,. Based on a predetermined operation of the operation input unit 4, the user-desired two main part equivalent areas A1 and A1 may be designated.

<Number of detections: 1>
When one main part equivalent area A1 is detected by the area detection unit 5d (step S17; one), the first main part setting part 5e uses the detected one main part equivalent area A1 as a mouth (main part). Is set as an operation processing target (step S191; see FIG. 6A). Specifically, the first main part setting unit 5e reads out and acquires the face main part image data 3c corresponding to the mouth from the storage unit 3, and the first main part image P1 representing the mouth is one main part equivalent region. The size and position of the first main part image P1 are set so as to be arranged in A1 (see FIG. 6B).

Next, the area specifying unit 5f specifies two main part corresponding areas A2 and A2 corresponding to the left and right eyes in the image I (step S192; see FIG. 6A). Specifically, the area specifying unit 5f uses two main part corresponding areas corresponding to both the left and right eyes in the image I based on the positional relationship, size, and the like in the image I of the one main part equivalent area A1. A2 and A2 are specified.
Subsequently, the second main part setting unit 5g sets the two specified main part corresponding areas A2 and A2 as the operation processing targets of the left and right eyes (other main parts) (step S193). Specifically, the second main part setting unit 5g reads out and acquires the face main part image data 3c corresponding to the left and right eyes from the storage unit 3, and each second main part image P2 representing both the left and right eyes is stored in two. The size and position of the second main part image P2 are set so as to be arranged in the two main part corresponding areas A2 and A2 (see FIG. 6B).

<Number of detections: 0>
When no region corresponding to the main portion A1 is detected by the region detection unit 5d (step S17; 0), the region detection unit 5d performs predetermined image processing (for example, edge processing) on the image data of the image I to be processed. Detection process or the like) is performed to detect the widest area where no edge exists from the image I as the face area C (step S201; see FIG. 7A).
Subsequently, in the detected face area C, the motion processing unit 5 considers the size of the face area C and the positional relationship of the main parts such as the left and right eyes and mouth, and the left and right eyes and mouth. The operation processing target of (main part) is set (step S202; refer to FIG. 7B). Specifically, the motion processing unit 5 sets the size and position of the main image P so that the main image P representing both the left and right eyes and the mouth is arranged in the face area C.

Returning to FIG. 2, the motion generation unit 5 h generates motion data for operating the main part (for example, both the left and right eyes and the mouth) in the image I (step S <b> 3).
Specifically, for example, when the number of detections in the main part equivalent area A1 is two (step S17; two), the motion generation unit 5h corresponds to both the left and right eyes set by the first main part setting unit 5e. The reference motion data 3b is read out and acquired from the storage unit 3, and the reference motion data 3b corresponding to the mouth set by the second main unit setting unit 5g is read out and acquired from the storage unit 3. Then, the motion generation unit 5h includes information indicating the size and position of the first main image P1 of the left and right eyes and the second main image P2 of the mouth in the image I to be processed, and the reference operation data 3b. The motion data of the first main part image P1 and the second main part image P2 is generated by associating with information indicating the movement of a plurality of control points in the prescribed space.
Further, for example, when the number of detections in the main part equivalent area A1 is one (step S17; one), the action generation unit 5h uses the reference action data 3b corresponding to the mouth set by the first main part setting unit 5e. While reading out and acquiring from the memory | storage part 3, the reference action data 3b corresponding to both the left and right eyes set by the 2nd main part setting part 5g is read out and acquired from the memory | storage part 3. FIG. Then, the motion generation unit 5h includes information indicating the size and position of the first main part image P1 of the mouth and the second main part image P2 of the left and right eyes in the image I to be processed, and the reference operation data 3b. The motion data of the first main part image P1 and the second main part image P2 is generated by associating with information indicating the movement of a plurality of control points in the prescribed space.
Further, for example, when the number of detections of the main part equivalent area A1 is 0 (zero) (step S17; 0), the motion generation unit 5h includes the left and right eyes and mouths in which the main part image P is set in the face area C. The reference motion data 3b corresponding to is read out from the storage unit 3 and acquired. The motion generation unit 5h includes a plurality of pieces of information in the predetermined space defined in the reference motion data 3b and information indicating the size and position of the left and right eyes and the main image P of the mouth in the image I to be processed. The motion data of the main part image P is generated in association with the information indicating the movement of the control point.

  When it is determined in step S14 that the main part of the face has been detected (step S14; YES) and the main part is extracted (step S15), the motion generation unit 5h extracts the extracted main part. The reference motion data 3b corresponding to is read out from the storage unit 3 and acquired. Then, the motion generation unit 5h sets a plurality of control points at predetermined positions in the main part extracted image Q, and corresponds to information indicating the movements of the plurality of control points in the predetermined space defined in the reference motion data 3b. In addition, operation data of the main part extracted image Q is generated.

  Further, the operation data generated by the operation generation unit 5h may be output to a predetermined storage unit (for example, the memory 2) and temporarily stored.

Next, the operation control unit 5i performs a process of operating the main part in the image I based on the operation data (step S4). Specifically, for example, the motion control unit 5i obtains motion data of motion data of the first main part image P1 and the second main part image P2 representing the main part and other main parts generated by the action generation unit 5h. The first main part image P1 and the second main part image P2 are acquired by displacing the plurality of control points based on the information indicating the movements of the plurality of control points defined in the acquired operation data. The operation process that operates is performed.
In addition, even when motion data of the main part image P or motion data of the main part extracted image Q is generated by the motion generation unit 5h, the motion control unit 5i is defined in the acquired motion data in a similar manner. By displacing the plurality of control points based on the information indicating the movement of the plurality of control points, an operation process for operating the main part image P and the main part extracted image Q is performed.

As described above, according to the image processing apparatus 100 of the present embodiment, a main part equivalent region (for example, a circle or a polygon in the image I) corresponding to the shape of the main part of the face in the acquired image I is formed. Area etc.) Since A1 is detected and the detected main part equivalent area A1 is set as an operation processing target of any main part, for example, image I that does not include a face, face detection or main part of the face Even if the image I with low detection accuracy is set as the processing target, the main part can be set in the image I to be processed, and the main part can be operated. Specifically, the first main part image P1 representing the main part is set to be superimposed on the main part equivalent area A1, and the first main part image P1 is deformed in the image I so that the movement is appropriate. Can be expressed in
In this way, the main part of the face can be set even for the image I that does not include the face.

Further, with reference to the position of the main part equivalent area A1 detected in the image I, the main part corresponding area A2 corresponding to the other main part is specified, and the specified main part corresponding area A2 is set to the other main part. It can be set as an operation processing target.
Specifically, for example, the main part corresponding to the mouth as the other main part on the basis of the positions of the two main part equivalent areas A1 and A1 corresponding to the left and right eyes detected as the main part in the image I Since the corresponding area A2 is specified, the main part corresponding area A2 corresponding to the mouth is specified even when only two main part corresponding areas A1 and A1 corresponding to both the left and right eyes in the image I are set. The specified main part corresponding area A2 can be set as a mouth motion processing target and the mouth can be operated. Further, since the main part corresponding area A2 corresponding to the eyes as the other main part is specified on the basis of the position of one main part corresponding area A1 corresponding to the mouth detected as the main part in the image I, the image Even when only the main part corresponding area A1 corresponding to the mouth is set in I, the main part corresponding area A2 corresponding to the eyes can be specified. The eye can be operated by setting it as an operation processing target.
At this time, the first main part image P1 representing the main part is set to be superimposed on the main part equivalent area A1, and the second main part image P2 representing the other main part is superimposed on the main part corresponding area A2. By setting and performing a deformation operation on the first main part image P1 and the second main part image P2 in the image I, the motion can be appropriately expressed.

In addition, when a face is not detected from the image I to be processed, the main part corresponding area A1 corresponding to the shape of the main part of the face is detected in the image I. Even if an image I with low detection accuracy of face detection is set as a processing target, an operation processing target of a main part can be set in the processing target image I according to the result of the face detection processing. That is, when a face is detected from the image I with high accuracy, the face can be used. On the other hand, when a face is not detected from the image I or when the face detection accuracy is low, By detecting the main part equivalent area A1, it is possible to set the detected main part equivalent area A1 as an operation processing target of the main part and operate the main part.
Similarly, when the main part of the face is not detected from the image I to be processed, the main part equivalent area A1 corresponding to the shape of the main part of the face is detected in the image I, so that the detection accuracy of the main part is detected. Even if the image I having a low value is set as the processing target, the operation processing target of the main part can be set in the processing target image I according to the result of the detection process of the main part. That is, when the main part of the face is detected from the image I with high accuracy, the main part can be used. On the other hand, when the detection accuracy of the main part is low, the main part equivalent area A1 in the image I is used. By detecting this, the detected main part equivalent area A1 can be set as an operation processing target of the main part and the main part can be operated.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
In the above embodiment, the image processing apparatus 100 is configured as a single unit. However, the image processing apparatus 100 is only an example, and the present invention is not limited thereto. For example, a projection target such as a person, a character, or an animal is a product or the like. The configuration may be applied to a projection system (not shown) that projects video content to be described on a screen.

Furthermore, in the above embodiment, the operation data for operating the main part and other main parts is generated. However, this is only an example, and the operation generation unit 5h is not necessarily limited to this. For example, information related to operation processing targets of the main part set in the image I by the first main part setting part 5e and the second main part setting part 5g and other main parts is not shown. The operation data may be generated by the external device.
Further, the main part and other main parts are operated. However, the main part and other main parts are only examples, and the present invention is not limited to this. The operation controller 5i is not necessarily provided. For example, the first main part setting unit 5e is used. Or the main part set in the image I by the second main part setting part 5g and information related to the operation processing target of the other main part are output to an external device (not shown), and the external device according to the control condition The main part or the entire face may be operated.

Further, in the above embodiment, when the main part of the face is detected from the image I to be processed, the operation is performed using the detected main part. The present invention is not limited, and even when the main part of the face is detected, it is not always necessary to use the main part.
That is, for example, it may be configured such that the user can select whether or not to use the detected main part based on a predetermined operation of the operation input unit 4. Further, the operation is performed using the detected main part in advance, or the main part equivalent area A1 is detected and the main part (first main part image P1) is set in the main part equivalent area A1. It is also possible to determine either of them. For example, the configuration may be selectable depending on whether the user's preference is to perform a deforming operation on a part that looks like a face in the image I or to perform a deforming operation on a part that does not look like a face in the image I.

  In the above embodiment, in the image I, two main part equivalent areas A1 and A1 and one main part corresponding area A2 are set as one set, or one main part corresponding area A1 and two main part corresponding areas A1. The movement of one face is expressed with the part corresponding areas A2 and A2 as one set, but is not limited to this example. For example, the main part corresponding area A1 and the main part corresponding area A2 A plurality of sets may be prepared, and a plurality of face motions may be expressed in the image I.

  In the above embodiment, when the main part equivalent area A1 in which the center of each of the three circular or polygonal shaped parts is arranged at the position of the apex of the triangle is detected, the left and right eyes and mouth When two main part equivalent areas A1 and A1 are detected, the left and right eyes are set, and when one main part equivalent area A1 is detected, it is set as a mouth. It is not limited. For example, when three or more main part equivalent areas A1,... Are detected, all of them are set to eyes only, and a mouth as a main part corresponding area A2 corresponding to each of them is set, or , All may be set to the mouth only, and the eye as the main part corresponding area A2 corresponding to each of them may be set.

  Furthermore, the configuration of the image processing apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto. For example, the image processing apparatus 100 may include a speaker (not shown) that outputs sound, and may output a predetermined sound from the speaker so as to perform lip sync during the process of operating the mouth in the image. . The audio data output at this time may be stored in association with the reference motion data 3b, for example.

  In the above embodiment, the main part and other main parts are exemplified as image processing, and the main part equivalent area A1 and the main part corresponding area A2 in the image I are set to the main part and other parts. However, the present invention is not limited to this example, and can be arbitrarily changed as appropriate to other image processing or the like. For example, an image representing a main part or another main part may be replaced with the main part corresponding area A1 or the main part corresponding area A2 to be a processing target of a replacement process in which only static display is performed.

In addition, in the above-described embodiment, the functions as the acquisition unit, the region detection unit, and the first setting unit are controlled under the control of the central control unit 1 of the image processing apparatus 100, the image acquisition unit 5a, the region detection unit. 5d is configured to be realized by driving the first main part setting unit 5e, but is not limited thereto, and is realized by executing a predetermined program or the like by the CPU of the central control unit 1. It is good also as a structure.
That is, a program including an acquisition process routine, an area detection process routine, and a setting process routine is stored in a program memory that stores the program. Then, the CPU of the central control unit 1 may function as a means for acquiring the image I by an acquisition processing routine. Further, the CPU of the central control unit 1 may function as means for detecting an area corresponding to the shape of the main part of the face (main part equivalent area A1) in the acquired image I by the area detection processing routine. good. Further, the CPU of the central control unit 1 may function as means for setting the detected area as an image processing target of any main part by the setting process routine.

  Similarly, the specifying unit, the second setting unit, the operation control unit, the face detection unit, and the main part detection unit may be realized by executing a predetermined program or the like by the CPU of the central control unit 1. .

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
An acquisition means for acquiring an image;
Area detecting means for detecting an area corresponding to the shape of the main part of the face in the image acquired by the acquiring means;
First setting means for setting the area detected by the area detection means as an image processing target of any main part;
An image processing apparatus comprising:
<Claim 2>
The image processing apparatus according to claim 1, wherein the area detecting unit further detects a circular or polygonal area in the image as an area corresponding to the shape of the main part.
<Claim 3>
A specifying unit for specifying a region corresponding to a main part other than the main part set by the first setting unit with reference to a position of the region detected by the region detection unit in the image;
The image processing apparatus according to claim 1, further comprising: a second setting unit that sets an area specified by the specifying unit as an image processing target of the other main part.
<Claim 4>
The first setting means further sets both the left and right eyes as the main part in two areas each forming a circle or a polygon detected by the area detection means in the image,
The said specifying means specifies the area | region corresponding to the mouth as said other main part on the basis of the position of the said 2 area | region detected by the said area | region detection means in the said image. An image processing apparatus according to 1.
<Claim 5>
The first setting means further sets a mouth as the main part in one area forming a circle or a polygon detected by the area detection means in the image,
4. The specifying unit specifies a region corresponding to an eye as the other main part on the basis of a position of the one region detected by the region detecting unit in the image. An image processing apparatus according to 1.
<Claim 6>
The said 1st setting means is further set to superimpose the image showing the said principal part on the area | region detected by the said area | region detection means, The Claim 1 characterized by the above-mentioned. Image processing device.
<Claim 7>
The image processing apparatus according to claim 6, further comprising an operation control unit that operates an image representing the main part set by the first setting unit in the image acquired by the acquisition unit.
<Claim 8>
The first setting unit is further configured to superimpose an image representing the main part on the region detected by the region detection unit,
The image processing apparatus according to claim 3, wherein the second setting unit is configured to superimpose an image representing the other main part on an area specified by the specifying unit.
<Claim 9>
Operation control means for operating the image representing the main part set by the first setting means and the image representing the other main part set by the second setting means in the image obtained by the obtaining means. The image processing apparatus according to claim 8, further comprising:
<Claim 10>
A face detection unit for detecting a face from the image acquired by the acquisition unit;
The area detection unit detects an area corresponding to a shape of a main part of a face in an image acquired by the acquisition unit when a face is not detected by the face detection unit. The image processing apparatus according to any one of 1 to 9.
<Claim 11>
A main part detecting unit for detecting a main part of the face from the image acquired by the acquiring unit;
The area detecting means detects an area corresponding to the shape of the main part of the face in the image acquired by the acquiring means when the main part is not detected by the main part detecting means. The image processing apparatus according to claim 1.
<Claim 12>
An image processing method using an image processing apparatus,
Processing to acquire images,
Processing for detecting an area corresponding to the shape of the main part of the face in the acquired image;
A process of setting the detected area as an image processing target of any main part;
An image processing method comprising:
<Claim 13>
The computer of the image processing device
Acquisition means for acquiring images,
A region detecting unit for detecting a region corresponding to the shape of the main part of the face in the image acquired by the acquiring unit;
First setting means for setting the area detected by the area detecting means as an image processing target of any main part;
A program characterized by functioning as

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 1 Central control part 5 Operation processing part 5a Image acquisition part 5b Face detection part 5c Face main part detection part 5d Area detection part 5e First main part setting part 5f Area specification part 5g Second main part setting part 5h Generation unit 5i Operation control unit

Claims (14)

An acquisition means for acquiring an image;
A region detecting unit for detecting a region corresponding to the shape of the main part of the face from a region not including the face image in the image acquired by the acquiring unit;
A first setting means for setting a region detected by the region detecting means as images processed,
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the first setting unit sets an area detected by the area detection unit as an image processing target related to a main part of the face. The image processing apparatus according to claim 1 , wherein the area detection unit further detects a circular or polygonal area in the image as an area corresponding to the shape of the main part. A specifying unit for specifying a region corresponding to a main part other than the main part set by the first setting unit with reference to a position of the region detected by the region detection unit in the image;
The image processing according to any one of claims 1 to 3, further comprising: a second setting unit that sets an area specified by the specifying unit as an image processing target of the other main part. apparatus. The first setting means further sets both the left and right eyes as the main part in two areas each forming a circle or a polygon detected by the area detection means in the image,
The specifying means, according to claim 4, characterized in that with reference to the position of the detected said two regions by the region detecting means in the image, specifying an area corresponding to the other mouth of the main portion An image processing apparatus according to 1. The first setting means further sets a mouth as the main part in one area forming a circle or a polygon detected by the area detection means in the image,
The specifying means, according to claim 4, characterized in that with reference to the position of the detected said one of regions by the region detecting means in the image, specifying an area corresponding to the eye as the other major portion An image processing apparatus according to 1. It said first setting means further according to any one of claims 1 to 6, characterized in that set so as to superimpose an image representing the principal part of the detected area by the area detection means Image processing device. The image processing apparatus according to claim 7 , further comprising an operation control unit that operates an image representing the main part set by the first setting unit in the image acquired by the acquisition unit. The first setting unit is further configured to superimpose an image representing the main part on the region detected by the region detection unit,
The image processing apparatus according to claim 4 , wherein the second setting unit is configured to superimpose an image representing the other main part on an area specified by the specifying unit. Operation control means for operating the image representing the main part set by the first setting means and the image representing the other main part set by the second setting means in the image obtained by the obtaining means. The image processing apparatus according to claim 9 , further comprising: A face detection unit for detecting a face from the image acquired by the acquisition unit;
It said region detecting means, claims, characterized in that the when the face is not detected by the face detecting means detects a region corresponding to the shape of the main part of a face in an image acquired by the acquisition unit The image processing apparatus according to any one of 1 to 10 . A main part detecting unit for detecting a main part of the face from the image acquired by the acquiring unit;
The area detecting means detects an area corresponding to the shape of the main part of the face in the image acquired by the acquiring means when the main part is not detected by the main part detecting means. The image processing apparatus according to any one of claims 1 to 11 . An image processing method using an image processing apparatus,
Processing to acquire images,
A process of detecting an area corresponding to the shape of the main part of the face from an area not including the face image in the acquired image;
A process of setting a detected region as images processed,
An image processing method comprising: The computer of the image processing device
Acquisition means for acquiring images,
Area detecting means for detecting an area corresponding to the shape of the main part of the face from an area not including the face image in the image acquired by the acquiring means;
First setting means for setting a detected region by the region detecting means as images processed,
A program characterized by functioning as

Images