JP6326808B2 - Face image processing apparatus, projection system, image processing method and program - Google Patents

Description

  The present invention relates to a face image processing apparatus, a projection system, an image processing method, and a program.

  2. Description of the Related Art Conventionally, a technique for deforming a face image based on a face area of a subject in a captured image and a control point for expressing an expression is known (see, for example, Patent Document 1). In addition, a method for detecting the degree of opening of the mouth from the luminance distribution of the mouth region in the face image (see, for example, Patent Document 2), and a method for detecting the degree of opening of the mouth from the area of the mouth region or tooth region in the face image (For example, refer to Patent Document 3).

JP 2012-185624 A JP 2009-231879 A JP 2005-234686 A

  By the way, the technique described in Patent Document 2 showing a method for detecting the degree of opening of the mouth may not be able to properly detect the degree of opening of the mouth because the luminance distribution changes depending on makeup or skin color. is there. Further, since the technique described in Patent Document 3 is for identifying facial expressions, it is difficult to determine whether the image is suitable for deformation of a face image.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a face image processing apparatus, a projection system, an image processing method, and a program capable of appropriately specifying the open / closed state of the mouth. Is to provide.

In order to solve the above problems, a face image processing apparatus according to the present invention is provided.
An acquisition means for acquiring a face image; a first detection means for detecting a mouth from the face image acquired by the acquisition means; and a region on the central side and a peripheral side of the mouth detected by the first detection means. A second detection unit that detects a degree of difference between color information in a predetermined color space of the region; and a specifying unit that specifies an open / closed state of the mouth based on a detection result by the second detection unit. It is characterized by that.

Moreover, the projection system according to the present invention includes:
A projection system comprising the face image processing device of the present invention and a projection device that projects a face image on a screen, wherein the face image processing device is identified as being closed by the specifying means. The image forming apparatus further includes a deforming unit that performs a deforming process on the image, and the projection device projects an image of the mouth subjected to the deforming process by the deforming unit onto the screen.

The image processing method according to the present invention includes:
An image processing method using a face image processing device, a process for acquiring a face image, a process for detecting a mouth from the acquired face image, a region on the center side and a region on the peripheral side of the detected mouth And a process for detecting the degree of difference between the color information in a predetermined color space, and a process for specifying the open / closed state of the mouth based on the detection result of the degree of difference between the color information.

The program according to the present invention is
The computer of the face image processing apparatus includes an acquisition unit for acquiring a face image, a first detection unit for detecting a mouth from the face image acquired by the acquisition unit, and a center side of the mouth detected by the first detection unit Second detecting means for detecting the difference between the color information in the predetermined color space of the area and the peripheral area, and specifying means for specifying the open / closed state of the mouth based on the detection result by the second detecting means It is characterized by functioning as.

  According to the present invention, the opening / closing state of the mouth can be specified appropriately.

It is a figure which shows typically the whole structure of the projection system of one Embodiment to which this invention is applied. It is a figure which shows typically the projection state by the projection system of FIG. It is a block diagram which shows the functional structure of the image processing apparatus which comprises the projection system of FIG. It is a flowchart which shows an example of the operation | movement which concerns on the projection process by the projection system of FIG. 6 is a flowchart illustrating an example of operations related to face image identification processing in the projection processing of FIG. 4. It is a figure which shows typically an example of the image which concerns on the face image specific process of FIG.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a diagram schematically showing an overall configuration of a projection system 100 according to an embodiment to which the present invention is applied. 2A is a front view of the projection system 100 schematically showing the non-projection state of the video content, and FIG. 2B is a diagram of the projection system 100 schematically showing the projection state of the video content. It is a front view.

As shown in FIG. 1, a projection system 100 according to this embodiment includes an image processing device 1, an imaging device 2, a projection device 3, a speaker 4, a screen 5, and the like. In addition, the projection system 100 projects, for example, video content on which a projection target such as a person, character, or animal explains a product or the like on the screen 5, and FIG. The positional relationship with each component of the projection system 100 is schematically represented.
The audience O may be one person or plural persons.

FIG. 3 is a block diagram illustrating a functional configuration of the image processing apparatus 1.
As shown in FIG. 3, the image processing apparatus 1 includes a central control unit 101, an operation input unit 102, a display unit 103, a recording unit 104, an image processing unit 105, an imaging control unit 106, and first to first images. The third I / F 107, 108, 109 and the like are provided. The central control unit 101, the display unit 103, the recording unit 104, the image processing unit 105, the imaging control unit 106, and the first to third I / Fs 107, 108, and 109 are connected via a bus line 110.
As the image processing apparatus 1, for example, a computer such as a personal computer or a workstation, a terminal device, or the like is applicable. Further, the image processing apparatus 1 may be configured to correct the projection image (frame image) of each frame of the video content so that the projected image of the video content projected on the screen 5 faces the audience O side.

  Although not shown, the central control unit 101 includes, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. The CPU of the central control unit 101 reads a program recorded in a predetermined recording area of the recording unit 104, develops it in the work area of the RAM, and executes various processes according to the expanded program. The RAM of the central control unit 101 is a volatile memory, and has a work area for storing various programs executed by the CPU, data related to these various programs, and the like.

  The operation input unit 102 is, for example, a keyboard or mouse configured by data input keys for inputting numerical values, characters, and the like, up / down / left / right movement keys for performing data selection, feeding operations, and various function keys. The operation unit is provided, and a predetermined operation signal is output to the central control unit 101 in accordance with the operation of these operation units.

  The display unit 103 is configured by, for example, an LCD (Liquid Crystal Display) or the like, and displays various types of information in a display area in accordance with a display control signal from the central control unit 101.

  The recording unit 104 includes, for example, an HDD (Hard Disk Drive), a semiconductor nonvolatile memory, or the like. The recording unit 104 records a system program executed by the central control unit 101, various processing programs, data necessary for executing these programs, and the like. For example, the program is stored in a predetermined recording area in the form of computer-readable program code.

The recording unit 104 records video data 104a of video content for projection.
The video data 104a is constituted by, for example, data of projection images (for example, images in which the projection target is facing the front direction) constituting the moving image data of the three-dimensional model and audio data corresponding to the projection images. Has been. A three-dimensional model is an image of a three-dimensional object composed of a three-dimensional polygon, a three-dimensional curved surface, a texture image, and the like.
In addition, the video data 104a includes motion information (for example, motion information of each face constituent unit) for expressing a motion of a projection target (particularly a human face) included in the video content. The movement information is information indicating the movement of a plurality of control points in a predetermined space. For example, information indicating the position coordinates (x, y) of the plurality of control points in the predetermined space, deformation vectors, and the like along the time axis. Are lined up.

The image processing unit 105 includes an image acquisition unit 105a, a face detection unit 105b, a mouth detection unit 105c, a first difference degree detection unit 105d, a second difference degree detection unit 105e, an open / close state specifying unit 105f, And a deformable portion 105g.
Note that each unit of the image processing unit 105 includes, for example, a predetermined logic circuit, but the configuration is an example and is not limited thereto.

The image acquisition unit (acquisition unit) 105a acquires an image including a face.
That is, for example, the image acquisition unit 105a acquires a copy of image data of an image including the face of the audience O captured by the imaging device 2 (for example, an image of only the face or an image above the chest). Specifically, the image acquisition unit 105a acquires a copy of the image data of the captured image captured by the imaging device 2 as a processing target image.
Further, when the imaging device 2 captures the face of the audience O again, the image acquisition unit 105a newly acquires a copy of the image data of the image including the face of the audience O captured by the imaging device 2.

  Note that the image acquisition unit 105a may read out and acquire an image recorded in the recording unit 104 after imaging the audience O by the imaging device 2. Each process by the image processing unit 105 to be described later may be performed on the image data itself of the processing target image, or a predetermined size (for example, VGA size) obtained by reducing the image data at a predetermined ratio as necessary. Etc.) may be performed on the reduced image data.

The face detection unit 105b detects a face area A1 from the processing target image.
That is, the face detection unit 105b performs a predetermined face detection process on the image data of the processing target image acquired by the image acquisition unit 105a, and selects a face region A1 (see FIG. 6A) including the face. To detect.
Since the face detection process is a known technique, detailed description thereof is omitted here.

The mouth detection unit (first detection unit) 105c detects the mouth P from the image including the face acquired as the processing target image by the image acquisition unit 105a (see FIG. 6A).
In other words, the mouth detection unit 105c uses, for example, each of the left and right eyes from the face area A1 in the processing target image detected by the face detection process using a predetermined detection process (for example, AAM (Active Appearance Model)). Detect major face components such as nose, mouth, eyebrows and face contours.
Here, AAM is a technique for modeling a visual event, and is a process for modeling an image of an arbitrary face region. For example, the mouth detection unit 105c prepares statistical analysis results of positions and pixel values (for example, luminance values) of predetermined feature parts (for example, eyes, nasal head, and face line) in a plurality of sample face images. A shape model representing the shape of the face and a texture model representing “Appearance” in the average shape are set on the basis of the position of the characteristic part, and a face image is modeled using these models. Thereby, in the face area A1, for example, face constituent parts such as left and right eyes, nose, mouth, eyebrows, and face contour are modeled.
As the detection process, for example, various processes such as an edge extraction process, an anisotropic diffusion process, and template matching may be used. In addition, the mouth detection unit 105c detects a main face component including the mouth P, but is not limited to this example. Any configuration may be used as long as it is configured to detect at least the mouth P. It can be changed.

The first difference degree detection unit (second detection unit) 105d detects the difference degree between the color information in the predetermined color space of the center side region and the peripheral side region of the mouth P detected by the mouth detection unit 105c. To do.
That is, the first difference detection unit 105d is, for example, a central region of a predetermined shape (for example, a substantially rectangular shape) arranged in a predetermined direction (for example, the vertical direction in FIG. 6B) so as to overlap the mouth P. An upper lip region B2 and a lower lip region B3 that sandwich B1 and the central region B1 in a predetermined direction are set. Then, the first dissimilarity detection unit 105d converts the image data in the central region B1, the upper lip region B2, and the lower lip region B3 into a predetermined color space (for example, HSV color space), and the predetermined color of each region. Generate a color map in space. Then, the first difference detection unit 105d performs a predetermined calculation based on the color map of the central region B1 and the color maps of the upper lip region B2 and the lower lip region B3, and performs the central region B1, the upper lip region B2, and the lower lip region. The degree of difference in the distribution region of the color map of B3 is detected as the first degree of difference.
In addition, in the center area B1, the upper lip area B2, and the lower lip area B3, the area outside the mouth P may be ignored.
In addition, as a method for detecting the degree of difference between color maps, for example, an arbitrary number of frequently used values including errors in the color map of each region, that is, representative values are extracted, and the values in each region are For example, a method of setting the sum of differences obtained by brute force and using the difference as a sum is used.

  In addition, although the upper lip region B2 and the lower lip region B3 sandwiching the central region B1 in a predetermined direction are illustrated as the peripheral side region of the mouth P, this is an example and not limited thereto. It can be arbitrarily arbitrarily changed to an annular region or the like surrounding the region B1.

The second difference degree detection unit (third detection means) 105e is a difference between the edge detection evaluation values of the region on the center side of the mouth P detected by the mouth detection unit 105c and the region adjacent to the region in the predetermined direction. Detect the degree.
That is, the second difference detection unit 105e performs, for example, predetermined edge detection processing (for the image data in the central region B1, the upper lip region B2, and the lower lip region B3 set by the first difference detection unit 105d ( For example, a predetermined matrix operation using a Hough transform or a Sobel filter is performed to detect the edge of each region. The second dissimilarity detection unit 105e performs a predetermined calculation based on the detection result of the edge of the central region B1 to calculate the central evaluation value related to the vertical edge, and the upper lip region B2 and the lower lip region B3. A predetermined calculation is performed based on the edge detection result to calculate the lip evaluation value related to the vertical edge, and the difference between the center evaluation value and the lip evaluation value is detected as the second difference.
The content of the edge detection process described above is an example and is not limited to this, and can be arbitrarily changed as appropriate. For example, instead of the Sobel filter, for example, a differential filter, a Prewitt filter, a Laplacian filter, or the like may be used, or any two or more of these may be used.

  In addition, as an area adjacent to the central area B1 of the mouth P in the predetermined direction, the upper lip area B2 and the lower lip area B3 sandwiching the central area B1 in the vertical direction are illustrated, but it is an example and not limited thereto. It can be arbitrarily changed as appropriate. Further, only one of the upper lip region B2 and the lower lip region B3 (for example, the upper lip region B2) may be used.

The open / closed state specifying unit (specifying unit) 105f specifies the open / closed state of the mouth P detected by the mouth detecting unit 105c.
That is, the open / closed state specifying unit 105f is based on the detection result of the degree of difference (first difference) between the color maps of the central region B1 of the mouth P, the upper lip region B2, and the lower lip region B3 by the first difference degree detection unit 105d. The opening / closing state of the mouth P is specified. Specifically, the opening / closing state specifying unit 105f determines whether or not the first difference detected by the first difference detection unit 105d is larger than a predetermined value, and as a result of the determination, the first difference is When it is determined that it is larger than the predetermined value, it is specified that the mouth P is in an open state.
When the mouth P is open, the teeth are seen from between the upper lip and the lower lip, and the difference between the color map of the central region B1 and the color map of the upper lip region B2 and the lower lip region B3 (first difference) This is because the degree is considered to increase.

Further, the open / closed state specifying unit 105f detects the degree of difference (second degree of difference) between the evaluation values of the edge detection of the center region B1 of the mouth P and the upper lip region B2 and the lower lip region B3 by the second difference degree detection unit 105e. Based on the result, the open / closed state of the mouth P is specified. Specifically, the open / closed state specifying unit 105f determines whether or not the second difference detected by the second difference detection unit 105e is greater than a predetermined value, and the second difference is determined as a result of the determination. When it is determined that it is larger than the predetermined value, it is specified that the mouth P is in an open state.
When the mouth P is open, the teeth are seen from between the upper lip and the lower lip, and the central evaluation value related to the vertical edge of the central region B1 corresponding to the teeth, the upper lip region B2, and the lower lip. This is because the degree of difference (second degree of difference) from the lip evaluation value relating to the vertical edge of the region B3 is considered to be large.
Here, instead of determining whether or not the difference between the central evaluation value and the lip evaluation value is larger than a predetermined value, the open / close state specifying unit 105f determines whether or not the central evaluation value is larger than the lip evaluation value. It may be determined that the mouth P is open when it is determined that the central evaluation value is larger than the lip evaluation value.

The image deformation unit (deformation unit) 105g performs a deformation process on the image of the mouth P identified as being closed by the open / close state identification unit 105f.
That is, the image deforming unit 105g sets a plurality of control points at predetermined positions of the mouth P detected by the mouth detecting unit 105c, and deforms the mouth P by displacing the positions of the set control points. Process.
Since the deformation process is a known technique, detailed description thereof is omitted here.

Further, the image deforming unit 105g synthesizes the image of the mouth P detected by the mouth detecting unit 105c at a position corresponding to the mouth of the face area of the projection target included in the video content projected from the projection device 3. For example, the image transformation unit 105g matches the corresponding position (for example, the center of gravity) of the mouth P detected by the mouth detection unit 105c with a predetermined position (for example, the center of gravity) of the mouth of the face area of the projection target. Thus, the image of the mouth P is replaced and synthesized. Then, the image deformation unit 105g performs a deformation process for deforming the mouth P based on movement information (for example, a deformation vector) of the mouth P associated with the video content.
Further, the image processing unit 105 outputs image data of an image including the mouth P subjected to the deformation process by the image deformation unit 105g to the projection device 3 together with the video data 104a.
As a method of combining the mouth P, Poisson image combining (Poisson image combining that generates an image in which the color and gradient of the boundary portion between the interpolation target region and other regions and the boundary portions of the plurality of replacement images are more naturally changed) (Poisson Image Editing) may be used, but this is an example, and the present invention is not limited to this, and can be arbitrarily changed.

The imaging control unit 106 controls imaging of the audience O by the imaging device 2. Specifically, the imaging control unit 106 includes a reimaging requesting unit (requesting unit) 106a that requests acquisition of a new face image.
When the opening / closing state specifying unit 105g specifies that the mouth P is open, the re-imaging request unit 106a sends a control signal for causing the imaging device 2 to image the face of the audience O again. Is output to the imaging device 2 via At this time, the re-imaging request unit 106a outputs a voice guidance with a predetermined content from the speaker 4 (for example, that the mouth P of the audience O is open or that the face of the audience O is re-imaged). Is output to the speaker 4 through the third I / F 109.

  The first I / F 107 is an interface that is connected to the imaging apparatus 2 and performs data transmission / reception with the imaging apparatus 2.

  The imaging device 2 is an imaging unit for imaging the audience O who is looking at the screen 5. The imaging device 2 is arranged immediately above the screen 5 so that the optical axis direction is perpendicular to the surface of the screen 5, and images the front direction of the screen 5 according to instructions from the image processing device 1. The obtained captured image is transmitted to the image processing apparatus 1 via the first I / F 107.

  The first I / F 108 is an interface that is connected to the projection apparatus 3 and performs data transmission / reception with the projection apparatus 3.

  The projection device 3 is, for example, a rear projection type projection device 3 that projects video content from the back of the screen 5 based on the projection image transmitted from the image processing device 1. For example, the projection device 3 can individually turn on / off each inclination angle of a plurality of micromirrors arranged in an array based on the projection image (for example, in the case of XGA, horizontal 1024 pixels × vertical 768 pixels). A DLP (Digital Light Processing) (registered trademark) projector using a DMD (digital micromirror device) which is a display element that forms a light image by reflected light by performing an off operation and a display operation is applicable. By turning on the micromirror at a position corresponding to the region of the projection object in the projection image and turning off the other regions, it is possible to project only the projection object on the screen 5.

  Further, the projection apparatus 3 receives the image data of the image including the mouth P and the video data 104a of the video content, which are output from the image processing unit 105 and subjected to the deformation process by the image deformation unit 105g, via the first I / F 108. Then, the video content is projected such that the mouth P synthesized in the face area of the projection target included in the video content is deformed (for example, lip sync) according to the content.

The screen 5 is supported by the support base 51 so as to be perpendicular to the floor surface, and is arranged in the output light irradiation direction of the projection device 3.
Further, as shown in FIG. 1, a screen film 53 for rear projection is pasted on the upper side of the front surface of a base material 52 such as a transparent acrylic plate molded into the shape of the projection target, and the screen 5 is projected on the lower side. A film 54 on which the lower half of the object is printed is attached.
In the present embodiment, the projection system 100 is configured to project the video content of the upper half of the projection target onto the screen 5, but is not limited to this example. A screen film 53 for rear projection may be attached to the entire surface of 52, and video content of the whole body of the projection target may be projected onto the screen 5.

  The third I / F 109 is an interface that is connected to the speaker 4 and performs data transmission / reception with the speaker 4.

  The speaker 4 converts predetermined audio data (for example, audio data included in the video data 104a) into an audio signal in response to an instruction output from the central control unit 101 and input via the third I / F 109. Output.

<Projection processing>
Next, projection processing by the projection system 100 will be described with reference to FIGS.
FIG. 4 is a flowchart illustrating an example of an operation related to the projection processing.

As shown in FIG. 4, first, the imaging device 2 captures the audience O to generate a live view image including a face (for example, an image of only the face or an image above the chest). The image acquisition unit 105a acquires a copy of the image data of the generated live view image (step S1).
Next, the face detection unit 105b performs a predetermined face detection process on the image data of the live view image acquired by the image acquisition unit 105a, and detects a face area A1 including the face (step S2). Subsequently, the image processing unit 105 determines whether or not the face area A1 is detected by the face detection unit 105b (step S3).

If it is determined in step S3 that the face area A1 is not detected (step S3; NO), the image processing unit 105 returns the process to step S2, and performs face detection processing at predetermined time intervals.
On the other hand, when it is determined in step S3 that the face area A1 has been detected (step S3; YES), the image processing unit 105 specifies a face image specifying process for specifying a face image to be combined with the projected video content ( (See FIG. 5).
Hereinafter, the face image specifying process will be described in detail with reference to FIG.

<Face image identification processing>
FIG. 5 is a flowchart illustrating an example of an operation related to the face image specifying process.
FIGS. 6A and 6B are diagrams schematically illustrating an example of an image related to the face image specifying process, and the image is not limited to these, and can be arbitrarily changed as appropriate.

As illustrated in FIG. 5, first, the imaging device 2 captures the audience O at a predetermined timing in accordance with voice guidance emitted from the speaker 4 to generate an image including a face, and the image acquisition unit 105 a of the image processing device 1. Acquires a copy of the generated image data of the captured image as a processing target image (step S11).
Next, the face detection unit 105b performs a predetermined face detection process on the image data of the processing target image acquired by the image acquisition unit 105a, and a face area A1 including the face (see FIG. 6A). Is detected (step S12).

  Subsequently, the mouth detection unit 105c detects the mouth P from the face area A1 in the processing target image detected by the face detection process using a predetermined detection process (for example, AAM) (step S13). At this time, together with the detection of the mouth P, for example, main face components such as the left and right eyes, the nose, the eyebrows, and the face contour may be detected.

  Next, the first difference detection unit 105d sets the central region B1, the upper lip region B2, and the lower lip region B3 so as to overlap with the mouth P (see FIG. 6B), and the central region B1, the upper lip region B2, The image data in the lower lip region B3 is converted into a predetermined color space (for example, HSV color space), and a color map in the predetermined color space of each region is generated (step S14). Then, the first dissimilarity detection unit 105d performs a predetermined calculation to detect the dissimilarity between the distribution areas of the color map of the central region B1, the upper lip region B2, and the lower lip region B3 as the first dissimilarity (step S15). .

Subsequently, the open / closed state specifying unit 105f determines the degree of difference (first difference) between the color maps of the center area B1, the upper lip area B2, and the lower lip area B3 of the mouth P detected by the first difference degree detection part 105d. It is determined whether or not it is larger than the predetermined value (step S16).
Here, when it is determined that the first degree of difference is greater than the predetermined value (step S16; YES), it is considered that the mouth P of the audience O is in an open state. A control signal for outputting voice guidance for prompting re-imaging of the face of the audience O is output to the speaker 4 via the first I / F 107. In response to the input control signal, the speaker 4 issues a voice guidance for prompting re-imaging of the face of the audience O (step S17).
Thereafter, in step S11, the imaging device 2 captures the audience O again at a predetermined timing in accordance with voice guidance emitted from the speaker 4 to generate an image including a face, and the image acquisition unit 105a generates the generated imaging. A copy of the image data of the image is acquired as a processing target image (step S11).

  Instead of the imaging device 2 imaging the face of the audience O again, the audience O is continuously imaged in advance to generate a plurality of images including the face, and the image acquisition unit 105 a A processing target image may be newly acquired from the image.

Each process after step S12 is repeatedly executed until it is determined in step S16 that the first difference is not greater than the predetermined value (step S16; NO).
When it is determined in step S16 that the first difference is not greater than the predetermined value (step S; YES), the second difference detection unit 105e is located in the central region B1, the upper lip region B2, and the lower lip region B3. The predetermined edge detection process is performed on the image data of the image area, and then a predetermined calculation is performed based on the detection result of the edge of the center area B1 to calculate the center evaluation value related to the vertical edge, and the upper lip area B2 Then, a predetermined calculation is performed based on the detection result of the edge of the lower lip region B3 to calculate the lip evaluation value related to the vertical edge (step S18). Then, the second difference degree detection unit 105e detects a difference degree between the center evaluation value related to the edge of the center area B1 and the lip evaluation values related to the edges of the upper lip area B2 and the lower lip area B3 as the second difference degree ( Step S19).

Subsequently, the open / closed state specifying unit 105f determines the difference (second difference) between the evaluation values of the edge detection of the center region B1, the upper lip region B2, and the lower lip region B3 detected by the second difference detection unit 105e. It is determined whether it is larger than a predetermined value (step S20).
Here, when it is determined that the second degree of difference is greater than the predetermined value (step S20; YES), since it is considered that the mouth P of the audience O is open, the process proceeds to step S17, and the process is repeated. In substantially the same manner as described above, the imaging request unit 106a outputs a control signal for outputting voice guidance for prompting re-imaging of the audience O's face from the speaker 4 to the speaker 4 via the first I / F 107. In response to the input control signal, the speaker 4 issues a voice guidance for prompting re-imaging of the face of the audience O (step S17).

Thereafter, the process returns to step S11, and each process after step S11 is repeatedly executed until it is determined in step S20 that the second difference is not greater than the predetermined value (step S20; NO).
If it is determined in step S20 that the second difference is not greater than the predetermined value (step S20; NO), the image deforming unit 105g sets a plurality of control points at the predetermined position of the detected mouth P. The plurality of control points are registered as control points used for the deformation process (step S21).
Thereby, the face image specifying process is completed.

  Returning to FIG. 4, the image processing unit 105 reads out and acquires the video data 104a of the specified video content from the recording unit 104 (step S5). Here, the video content is specified in advance by the CPU of the central control unit 101 based on a predetermined operation of the operation input unit 102 by the user. For example, the audience O can select the video content. A plurality of video contents may be prepared and the audience O may select and specify desired video contents.

Next, the image deforming unit 105g generates image data of the image obtained by deforming the mouth P by the deformation process, and the image processing unit 105 uses the image data generated by the image deforming unit 105g together with the video data 104a of the video content. And output to the projection apparatus 3 via the first I / F 108 (step S6).
Then, the projection device 3 is based on the video data 104a of the video content input via the first I / F 108 and the image data generated by the image transformation unit 105g within the face area of the projection target included in the video content. The video content that deforms (lip sync) the mouth P synthesized in accordance with the content is projected onto the screen 5 (step S7).
At this time, the image deforming unit 105g generates image data of an image obtained by deforming the face constituent unit other than the mouth P by the deformation processing, and the projection device 3 uses the projection data included in the video content based on the image data. You may project on the screen 5 the video content which deform | transforms face structure parts other than the mouth P within the face area of a thing.

Next, in the image processing device 1, the CPU of the central control unit 101 determines whether or not the projection of the video content by the projection device 3 has been completed (step S8).
Here, when it is determined that the projection of the video content has not ended (step S8; NO), the CPU of the central control unit 101 performs the determination process at predetermined time intervals.
On the other hand, when it is determined that the projection of the video content has ended (step S8; YES), the CPU of the central control unit 101 ends the projection process.

As described above, according to the projection system 100 of the present embodiment, the area on the center side (center area B1) and the area on the peripheral side (upper lip area B2 and lower lip area B3) of the mouth P detected from the face image. Since the degree of difference between the color information in the predetermined color space is detected and the open / closed state of the mouth P is specified based on the detection result, the color information in the predetermined color space in the central area and the peripheral area is obtained. The opening / closing state of the mouth P can be specified appropriately. That is, when the color information in the predetermined color space is different between the center side region and the peripheral side region of the mouth P, it is considered that the mouth P is opened by looking into the teeth between the upper lip and the lower lip. . Therefore, when the degree of difference between the color information in the predetermined color space in the central area and the peripheral area of the mouth P is large, the mouth P is specified to be in an open state. It can be specified appropriately.
Thereby, the deformation process for the mouth P can be performed using the image of the mouth P in the closed state. For example, when the image of the mouth P subjected to the deformation process is projected on the screen 5, Lip sync can be made more natural.

  Further, the degree of difference between the evaluation values of the edge detection of the region on the center side of the mouth P (central region B1) and the region adjacent to the region in the predetermined direction (upper lip region B2 and lower lip region B3) is detected. Since the opening / closing state of the mouth P is specified based on the result, the opening / closing state of the mouth P is specified using the edge detection evaluation values of the central side region and the adjacent region in addition to the color information of the mouth P. Can be performed more appropriately. That is, when the edge detection evaluation value is different between the region on the center side of the mouth P and the region adjacent to the region, it is considered that the mouth P is opened through the teeth between the upper lip and the lower lip. . Therefore, when the degree of difference between the edge detection evaluation values in the central area and the adjacent area is large, it is specified that the mouth P is in an open state. Can be identified.

  Further, when it is specified that the mouth P is in an open state, a new face image to be processed can be acquired in response to the request by requesting acquisition of a new face image. As a result, it is possible to prevent the deformation process from being performed using the image of the mouth P in the open state. As a result, when the image of the mouth P subjected to the deformation process is projected onto the screen 5. It is possible to suppress the lip sync from becoming unnatural.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, in the above-described embodiment, the projection system 100 that projects an image on the screen 5 is illustrated. However, the projection system 100 is merely an example, and the projection system 100 is not limited thereto. May be.

In the above embodiment, the image processing apparatus 1 opens and closes the mouth P according to the degree of difference (second degree of difference) between the edge detection evaluation values of the central area B1, the upper lip area B2, and the lower lip area B3. However, the present invention is not limited to this example, and the second difference degree detection unit 105e that detects the second difference degree is not necessarily provided.
Further, the image processing apparatus 1 performs the deformation process on the image of the mouth P in the closed state. However, the image processing apparatus 1 is an example, and is not limited thereto. It is not necessary to include the part 105g.

In addition, in the above embodiment, the functions of the acquisition unit, the first detection unit, the second detection unit, and the identification unit are acquired under the control of the central control unit 101 of the image processing apparatus 1. The unit 105a, the mouth detection unit 105c, the first dissimilarity detection unit 105d, and the open / close state specifying unit 105f are driven. However, the present invention is not limited to this, and the CPU of the central control unit 101 performs predetermined processing. It is good also as a structure implement | achieved by executing this program.
That is, a program including an acquisition processing routine, a first detection processing routine, a second detection processing routine, and a specific processing routine is stored in a program memory that stores the program. Then, the CPU of the central control unit 101 may function as a means for acquiring a face image by an acquisition process routine. Further, the CPU of the central control unit 101 may function as a means for detecting the mouth P from the acquired face image by the first detection processing routine. Further, the CPU of the central control unit 101 functions as a means for detecting the degree of difference between the color information in the predetermined color space of the central area and the peripheral area of the detected mouth P by the second detection processing routine. You may make it let it. Further, the CPU of the central control unit 101 may function as means for specifying the open / closed state of the mouth P based on the detection result of the degree of difference between the color information by the specifying process routine.

  Similarly, the erasing unit, the third detecting unit, the requesting unit, and the deforming unit may be realized by executing a predetermined program or the like by the CPU of the central control unit 101.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
Acquisition means for acquiring a face image;
First detection means for detecting a mouth from the face image acquired by the acquisition means;
Second detection means for detecting a degree of difference between color information in a predetermined color space in the central area and the peripheral area of the mouth detected by the first detection means;
Based on the detection result by the second detection means, specifying means for specifying the open / closed state of the mouth,
A face image processing apparatus comprising:
<Claim 2>
The specifying means further includes:
The face image processing apparatus according to claim 1, wherein when the degree of difference detected by the second detection unit is large, the face image processing device is identified as having an open mouth.
<Claim 3>
A third detection means for detecting the difference between the evaluation value of the edge detection of the area on the center side of the mouth detected by the first detection means and the area adjacent to the area in a predetermined direction;
The specifying means is:
The face image processing apparatus according to claim 1, wherein the opening / closing state of the mouth is specified based on a detection result of the third detection unit.
<Claim 4>
The specifying means further includes:
The face image processing apparatus according to claim 3, wherein when the degree of difference detected by the third detection unit is large, the face image processing device is identified as having an open mouth.
<Claim 5>
When it is specified by the specifying means that the mouth is in an open state, it further comprises request means for requesting acquisition of a new face image,
The acquisition means includes
The face image processing apparatus according to claim 1, wherein a face image is newly acquired in response to a request from the request unit.
<Claim 6>
A projection system comprising: the face image processing apparatus according to claim 1; and a projection apparatus that projects a face image on a screen.
The face image processing device includes:
Deformation means for performing deformation processing on the mouth image identified as being closed by the identification means,
The projector is
A projection system, wherein an image of a mouth subjected to deformation processing by the deformation means is projected onto the screen.
<Claim 7>
An image processing method using a face image processing device,
Processing to acquire a face image;
Processing to detect the mouth from the acquired face image;
A process for detecting the degree of difference between the color information in a predetermined color space in the center side area and the peripheral side area of the detected mouth;
Based on the detection result of the degree of difference between the color information, a process of specifying the open / closed state of the mouth,
An image processing method comprising:
<Claim 8>
The computer of the face image processing device
Acquisition means for acquiring a face image;
First detection means for detecting a mouth from the face image acquired by the acquisition means;
Second detection means for detecting the degree of difference between the color information in a predetermined color space of the central side area and the peripheral side area of the mouth detected by the first detection means;
Identification means for identifying the open / closed state of the mouth based on the detection result by the second detection means;
A program characterized by functioning as

DESCRIPTION OF SYMBOLS 100 Projection system 1 Image processing apparatus 101 Central control part 105 Image processing part 105a Image acquisition part 105c Mouth detection part 105d 1st difference detection part 105e 2nd difference detection part 105f Opening / closing state specific | specification part 105g Image deformation part 106 Imaging control part 106a Re-imaging request unit 2 Imaging device 3 Projection device

Claims (9)

Acquisition means for acquiring a face image;
First detection means for detecting a mouth from the face image acquired by the acquisition means;
Second detection means for detecting a degree of difference between color information in a predetermined color space in the central area and the peripheral area of the mouth detected by the first detection means;
Based on the detection result by the second detection means, specifying means for specifying the open / closed state of the mouth,
A face image processing apparatus comprising: The specifying means further includes:
The face image processing apparatus according to claim 1, wherein when the degree of difference detected by the second detection unit is large, the face image processing device is identified as having an open mouth. A third detection means for detecting the difference between the evaluation value of the edge detection of the area on the center side of the mouth detected by the first detection means and the area adjacent to the area in a predetermined direction;
The specifying means is:
The face image processing apparatus according to claim 1, wherein the opening / closing state of the mouth is specified based on a detection result of the third detection unit. The specifying means further includes:
The face image processing apparatus according to claim 3, wherein when the degree of difference detected by the third detection unit is large, the face image processing device is identified as having an open mouth. When it is specified by the specifying means that the mouth is in an open state, it further comprises request means for requesting acquisition of a new face image,
The acquisition means includes
The face image processing apparatus according to claim 1, wherein a face image is newly acquired in response to a request from the request unit. A projection system comprising: the face image processing apparatus according to claim 1; and a projection apparatus that projects a face image on a screen.
The face image processing device includes:
Deformation means for performing deformation processing on the mouth image identified as being closed by the identification means,
The projector is
A projection system, wherein an image of a mouth subjected to deformation processing by the deformation means is projected onto the screen. An image processing method using a face image processing device,
Processing to acquire a face image;
Processing to detect the mouth from the acquired face image;
A process for detecting the degree of difference between the color information in a predetermined color space in the center side area and the peripheral side area of the detected mouth;
Based on the detection result of the degree of difference between the color information, a process of specifying the open / closed state of the mouth,
An image processing method comprising: The computer of the face image processing device
Acquisition means for acquiring a face image;
First detection means for detecting a mouth from the face image acquired by the acquisition means;
Second detection means for detecting the degree of difference between the color information in a predetermined color space of the central side area and the peripheral side area of the mouth detected by the first detection means;
Identification means for identifying the open / closed state of the mouth based on the detection result by the second detection means;
A program characterized by functioning as Acquisition means for acquiring a face image;
  First detection means for detecting a mouth from the face image acquired by the acquisition means;
  Second detection means for detecting a degree of difference between color information in a predetermined color space in the central area and the peripheral area of the mouth detected by the first detection means;
  A specifying means for specifying that the mouth is open when the degree of difference detected by the second detecting means is large;
  A face image processing apparatus comprising:

Images