JP5951966B2 - Image processing apparatus, image processing system, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program for recognizing a motion of a feature portion of a user's face.

  Conventionally, for example, as shown in FIG. 8, the user's face is imaged by an imaging means such as a camera 210, and the movement of the nose, mouth, etc. in the face is recognized based on the captured image, and the movement Accordingly, a method of moving an image (pointer or the like) 218 displayed on the display screen 216 has been proposed (see, for example, Patent Document 1 and Patent Document 2).

JP-A-7-93089 JP 2009-31368 A

  However, according to the conventional technique, when the user changes his / her posture while moving the pointer 218 (not for the purpose of moving the image 218), the pointer 218 moves, which causes inconvenience. There was a thing.

  Specifically, for example, when the user corrects his / her posture and takes a posture that sits deeply on the chair and sinks to the lower left, the nose or the like to be recognized moves due to the posture change. Therefore, the pointer 218 moves to the left end of the display screen 216 along with the movement, and the user moves the face by the usual amount in an attempt to move the pointer 218 to the upper right of the display screen 216 while maintaining the posture. There has been a problem that the user does not reach the intended position or the user loses sight of the pointer 218.

  The present invention has been proposed in order to solve the above-described problem, and even when the user moves the face and moves the image on the display screen, the user can change the posture or the like. An image processing apparatus, an image processing system, an image processing method, and a program that can easily arrange the image at a reference position on a display screen and can easily continue the movement of the image even after the posture change is provided. For the purpose.

In order to achieve the above object, an image processing apparatus according to the present invention includes a feature detection unit that detects a feature of a user's face from a captured image obtained by capturing the user, and the user's face from the captured image. Direction detecting means for detecting the orientation of the camera, and whenever the predetermined face orientation is detected by the orientation detecting means, the feature detection means detects when the predetermined face orientation is detected. features detected by been a storage processing means for storing the position of the feature to identifiably storing means as the latest reference points, the feature detecting unit relative to the latest reference point stored in said storage means First control means for controlling the display of the image so that the predetermined image displayed on the display screen moves based on the relative position of the part, and the predetermined face orientation by the orientation detection means Detected To come, and a, and a second control means for controlling the display of the image as the image to a predetermined reference position is the arrangement of the display screen.

The image processing system according to the present invention includes an imaging unit that captures an image of a user, a feature detection unit that detects a feature portion of the user's face from the captured image captured by the imaging unit, and the captured image. The direction detecting means for detecting the orientation of the user's face, and the feature when the predetermined face orientation is detected every time the orientation detecting means detects the predetermined face orientation. a storage processing means for identifiably stored in the storage means the position of the feature portion detected by the section detecting means as the latest reference points, the feature detecting unit relative to the latest reference point stored in said storage means First control means for controlling the display of the predetermined image displayed on the display screen based on the relative position of the characteristic part detected by the above-mentioned, and the predetermined by the orientation detection means. Et And when the direction is detected face, and a, and a second control means for controlling the display of the image such that the image at a predetermined reference position is the arrangement of the display screen.

In addition, the image processing method of the present invention detects a feature portion of a user's face from a captured image obtained by capturing the user, detects the orientation of the user's face, and determines the user's face from the captured image. Each time a predetermined face orientation is detected, the position of the feature detected when the predetermined face orientation is detected is stored in the storage means so as to be identifiable as the latest reference point. Based on the relative position of the feature portion with respect to the latest reference point stored in the storage means, the display of the image is controlled so that the predetermined image displayed on the display screen moves, When a predetermined face orientation is detected, the display of the image is controlled so that the image is arranged at a predetermined reference position on the display screen.

Further, the program of the present invention includes a computer that detects a feature portion of a user's face from a captured image obtained by capturing the user, and a direction that detects the orientation of the user's face from the captured image. Each time a predetermined face orientation is detected by the detection means and the orientation detection means, the position of the feature detected by the feature detection means when the predetermined face orientation is detected is detected. storage processing means for storing the identifiable storing means as the latest reference point, based on the relative positions of the features detected by the feature detecting means relative to the latest reference point stored in said storage means, When a predetermined face orientation is detected by the first control means for controlling the display of the image so that the predetermined image displayed on the display screen moves, and the orientation detection means, Is a program for functioning as a second control means, for controlling the display of the image such that the image is located at a predetermined reference position of the display screen.

  As described above, according to the present invention, when the user moves an image on the display screen by moving his / her face, even if the user changes the posture, the reference of the display screen can be easily obtained. There is an effect that the image can be arranged at the position, and the movement of the image can be easily continued even after the posture change.

1 is a diagram illustrating an image processing system according to an embodiment, a processing device connected to the image processing system, and a monitor. It is a figure which shows an example of a structure of an image processing apparatus. It is an example of the flowchart which shows the flow of the face detection process performed with an image processing apparatus. It is explanatory drawing explaining a face and face part detection. It is a figure which shows typically the display position of the pointer according to the detection result while showing typically the detected user's face part field and face field. (A) is a figure which shows the display control example of a pointer when moving the whole face in the state which inclined the face, (B) is the case when moving the whole face in the state where the face turned to the front It is a figure which shows the example of a display control of a pointer. It is a figure which shows a mode that the display screen of a monitor is comprised by several screen area | regions, and the reference position is set separately to each of several screen area | regions. It is explanatory drawing for demonstrating the prior art which moves a pointer by a user moving a face.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows an image processing system 1 according to the present embodiment, a processing device 14 connected to the image processing system 1, and a monitor 16.

  The image processing system 1 includes a camera 10 and an image processing device 12. In the image processing system 1, first, the camera 10 captures the user's face as the recognition object 30 (in FIG. 1, the user's back head is illustrated, but in the present embodiment, the camera 10 is directed to the camera 10. And the user's face is detected from the image captured by the image processing device 12, and the detection result is output to the processing device 14. Based on the output detection result, the processing device 14 moves an image (a pointer in the present embodiment) 18 displayed on the display screen 16a of the monitor 16 serving as a display unit. The processing device 14 may be a personal computer, for example.

  FIG. 2 shows the configuration of the image processing apparatus 12.

  The image processing apparatus 12 includes a central processing unit (CPU) 20, a read only memory (ROM) 22, and a random access memory (RAM) 24, which are connected to each other via a bus (not shown). It is composed of a general computer. Furthermore, the image processing apparatus 12 includes an image processing memory 26.

  The program stored in the ROM 22 is loaded into the RAM 24 when the CPU 20 is started up, and the CPU 20 executes the program stored in the ROM 22 to control the entire image processing apparatus 12 and to perform face detection processing described later (see also FIG. 3). .) Is configured to perform. The ROM 22 may be configured by a flash memory. The storage medium for storing programs executed by the CPU 20 is not limited to the ROM 22. For example, it may be an HDD (Hard Disk Drive), a removable USB memory, or a storage device connected to a communication network.

  In the image processing memory 26, image data representing a captured image captured by the camera 10 is temporarily stored. The CPU 20 performs image processing on the image data stored in the image processing memory 26 and outputs a processing result (detection result) to the processing device 14.

  Although not shown, the processing device 14 includes a CPU, a RAM, a ROM, and designation means for designating various settings. For example, the system administrator designates various settings such as the size of the display screen 16a of the monitor 16 and the accuracy of face detection by the designation means, and the designated settings are input from the processing device 14 to the image processing device 12. . The image processing device 12 operates according to the input settings. For example, the scale of the coordinate value of the recognition result output to the processing device 14 may be adjusted based on the input information about the size of the display screen 16a. In addition, the accuracy information indicating the accuracy of the input face detection is adjusted to the resolution at the time of face detection or limited to the face size to be detected depending on, for example, whether priority is given to accuracy or priority to speed. May be provided. Note that these settings are merely examples, and the present invention is not limited to these.

  Note that image data of a captured image captured by the camera 10 from the image processing apparatus 12 may be input to the processing apparatus 14. As a result, the monitor 16 can also display the captured image captured by the camera 10. The captured image may be displayed on the display screen 16a of the monitor 16 or may not be displayed.

  The camera 10 that captures a captured image includes, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), and the like, and is configured to be able to continuously capture a subject. Image data representing the captured image is sequentially input to the image processing device 12 and temporarily stored in the image processing memory 26.

  FIG. 3 is an example of a flowchart showing the flow of face detection processing performed by the image processing apparatus 12. This process is started when the function of the system is started, such as when a face detection process start command is input from the processing device 14. Until the following step 116 is started and the user moves his / her face, the predetermined reference position of the display screen 16a of the monitor 16 (in this embodiment, the coordinate position of the center of the display screen 16a of the monitor 16) is set. It is assumed that the state where the pointer 18 is displayed continues. In the following description, the simple “position” indicates the absolute position in the XY coordinate system, and the “position relative to some point” or the “relative position” indicates the XY coordinate system. The relative position with respect to some point shall be indicated.

  In step 100, image data input from the camera 10 is read from the image processing memory 26, and image processing is performed on the image data, thereby predicting a face that is predicted to be present from the captured image represented by the image data. An area 40 is obtained (face area prediction). The captured image processed here is referred to as an initial image.

  Here, the position and size of the face are predicted using Hue (hue), Sat (saturation), and Lum (luminance). Specifically, for each of hue, saturation, and luminance, a frequency distribution of pixels having pixel values (hue value, saturation value, luminance value) corresponding to a human face (skin color) included in the captured image is shown. A histogram is generated in each of the X-axis and Y-axis directions, and a logical sum of the hue, saturation, and luminance histograms is calculated to obtain a region where a human face is predicted to exist. By such processing, as shown in FIG. 4A, the predicted face detection region 40 is determined for the recognition target object 30.

  Subsequently, in step 102, a predicted face part region 42 that is predicted to have eyes, nose, and mouth (hereinafter referred to as face parts) is obtained from the predicted face detection region 40 (face part region prediction). This is because the approximate relative position of each face part relative to the face is determined, such as a human face with eyes on the left and right of the top, a nose near the center, and a mouth at the bottom. The relative position information is stored in the ROM 22 in advance, and prediction is performed according to the position information and the size of the face in the predicted face detection area 40. As a result, as shown in FIG. 4B, a predicted face part area 42 corresponding to the eyes, nose, and mouth is obtained for the predicted face detection area 40. Here, the eyes, nose, and mouth will be described as specific examples of face parts, but the present invention is not limited to this. For example, eyebrows may be included as other face parts.

  In step 104, each face part (face part area 44) is detected from the predicted face part area 42 in the initial image. For example, as in step 100, when detecting eyes, in the predicted face part region 42 where it is predicted that there are eyes, pixels corresponding to the eyes included in the captured image for each of hue, saturation, and luminance A histogram showing the frequency distribution of pixels having values (hue value, saturation value, luminance value) is generated in each direction of the X axis and Y axis, and the logical sum of the histograms of hue, saturation, and luminance is obtained. Thus, an area where the eyes are predicted to exist is obtained. Alternatively, a method such as pattern matching may be used. As a result, as shown in FIG. 4C, a face part region 44 is detected for each face part.

  In step 106, it is determined whether or not all face parts have been detected from the initial image. If a negative determination is made here, it is determined that the user's face is not facing the front with respect to the camera 10, so the process returns to step 100 again and the above processing is repeated.

  If the determination in step 106 is affirmative, it is determined that each of the facial parts has been detected in a state where the user's face is facing the front of the camera 10, so the process proceeds to step 108, and the detected facial part region 44 is detected. Is stored in the RAM 24 and set. The area information includes information indicating the position and information indicating the size. This area information is sequentially updated every time there is a change during tracking described later.

  Subsequently, in step 110, the face area 46 is set based on each of the set face part areas 44. As mentioned above, the approximate relative position of each face part relative to the face, such as the eyes on the left and right of the upper part of the face, the nose near the center, and the mouth at the lower part of the face, is determined. In accordance with the position information, an accurate face area is obtained from the face part area 44, contrary to the step 102. Then, area information indicating the detected face area 46 is stored in the RAM 24 and set. This area information is also updated whenever there is a change during tracking described later.

  In step 112, among the plurality of face part areas 44 detected from the initial image, the position of a predetermined face part area 44 (in this embodiment, the nose area) (in this embodiment, the center of gravity is determined). Position) is a reference point 48, and the position information of the reference point 48 is stored in the RAM 24 and set. The nose region corresponds to the facial feature of the present invention.

  Here, the nose region is taken as an example of the facial feature, but the feature may be a mouth region, a left-eye region, or a right-eye region. Further, it may be an area sandwiched between the left eye area and the right eye area (for example, an area including an intermediate point between the left end of the left eye area and the right end of the right eye area or the intermediate point itself).

  In addition, the position of the nose region is the center of gravity of the nose region, but is not limited to this. For example, it may be the position of the upper left end of the nose region, the position of the lower right end, or the nose. It is good also as a position of the middle point of two outer nostrils.

  In step 114, information on the relative position of the set reference point 48 with respect to the face area 46 (hereinafter referred to as reference point relative position information) is stored in the RAM 24 and held. When the face part regions 44 of all the face parts are detected from the initial image, it is considered that the user is facing the front with respect to the camera 10. Therefore, in the following, the relative position of the reference point 48 is determined. The user is assumed to be a relative position in a state where the user faces the camera 10.

  As a result, the reference point update region 50 is determined (see also FIG. 6). In the present embodiment, the reference point update area 50 is a square area having a predetermined size with the position determined from the relative position indicated by the reference point relative position information with respect to the latest detected face area 46 as the center of gravity position. It is. The reference point update area 50 is an area used to determine whether or not the face is facing the front with respect to the camera 10, but will be described later when it is detected that the face is facing the front. Since the reference point 48 is updated as described above, this region is referred to as a reference point update region.

  In step 116, face parts and face tracking are started. Here, the face part and face tracking refers to detecting and tracking the detected face part region 44 and the face region 46 from each of the images photographed continuously or intermittently by the camera 10. Say. If the user's face moves, the positions of the face part area 44 and the face area 46 move accordingly. The tracking method is not particularly limited. For example, the face part region 44 and the face region 46 may be tracked by performing a well-known contour tracking process, or may be tracked by repeatedly performing the processes of step 104 and step 110. .

  In step 118, based on the face part area 44 and the face area 46 detected during the tracking period, it is determined whether or not the face is facing the front. A specific example of this determination method will be described later. If it is determined that the face is not facing the front, the process proceeds to step 120, and it is determined whether or not any of the face part areas 44 is not detected. If it is determined that any region of the face parts (left eye, right eye, nose, mouth) is not detected, the user's face is lost and tracking is stopped in step 124. Returning to 100, the user's face area and face parts are detected again. At this time, if the display position of the pointer 18 is not the reference position, control is performed so as to return to the reference position.

  In the present embodiment, tracking only the nose of the facial parts may cause the nose region to be erroneously detected when a person passes between the user and the camera 10. In the form, each face part is detected, and when at least one face part is not detected, the process returns to step 100.

  However, for example, in an operation in which the possibility of erroneous detection is low (for example, a person does not pass between the user and the camera 10), “at least two” or “at least three” are detected. If it is not performed, the process may return to step 100. Further, if only the face part area 44 (here, the nose area) used for operating the pointer 18 is detected among the face parts, the pointer 18 can be operated continuously without returning to step 100. You may comprise.

  On the other hand, if a negative determination is made in step 120, the process proceeds to step 124, and the position of the nose region (relative position of the nose region with respect to the reference point 48) is determined with reference to the latest reference point 48 stored in the RAM 24. , Information indicating the relative position (hereinafter, relative position information) is input to the processing device 14. Here, the position of the nose region is set as the barycentric position of the nose region, but it is needless to say that the position is not limited to the barycentric position as described above. The processing device 14 moves the pointer 18 to a coordinate position corresponding to the input relative position information on the display screen 16a. As described above, the image processing device 12 controls the pointer 18 displayed on the display screen 16 a of the monitor 16 to move by inputting the relative position information to the processing device 14.

  Here, display control of the pointer 18 by tracking will be described with reference to FIG. FIG. 5 is a diagram schematically showing the detected face part area 44 and the face area 46 of the user, and schematically showing the display result of the pointer 18 according to the detection result.

  In the initial stage where the user is facing the front and the face is not yet moved, the pointer 18 is arranged at the reference position of the display screen 16a of the monitor 16 as shown in FIG.

  Here, it is assumed that the user rotates his / her face to the right with respect to the camera 10 to change the face orientation. As a result, the position 48a of the nose region moves as shown in FIG. The image processing device 12 inputs relative position information corresponding to this movement to the processing device 14, and the processing device 14 responds to the relative position information from the reference position of the display screen 16a according to the relative position information. The pointer 18 is moved to the right position by the distance.

  When the user faces upward, the nose region position 48b moves as shown in FIG. The image processing device 12 inputs relative position information corresponding to this movement to the processing device 14, and the processing device 14 responds to the relative position information from the reference position of the display screen 16a according to the relative position information. The pointer 18 is moved to an upper position by the distance.

  In the present embodiment, the pointer 18 can be moved simply by changing the orientation of the face, but the nose region can also be moved by moving the entire face with the face tilted. In the latter case, the nose region moves due to both the movement of the entire face and the inclination of the face, so that the pointer 18 can be moved by the movement amount obtained by adding the movement amounts of the respective movements. Therefore, when a user performs a fine pointer operation or a pointer operation with a small amount of movement, the user may perform the pointer operation by tilting the face from a front direction to a different direction, or perform a pointer operation with a large amount of movement. When performing, the pointer operation may be performed by moving the entire face with the face tilted in the different direction.

  After step 124, the process returns to step 118.

  On the other hand, if it is determined in step 118 that the face is facing the front, the process proceeds to step 126, where the position information of the current center of gravity of the nose region is stored and set in the RAM 24 as the reference point 48. Here, the position information of the reference point 48 of the RAM 24 may be overwritten and updated, or may be stored in a storage area different from the already stored position information. However, in the latter case, it is necessary to configure the latest reference point 48 so that it can be identified by setting a flag on the position information of the latest reference point 48.

  In step 128, the position of the pointer 18 is reset. Here, coordinate information (or a reset command) corresponding to the reference position is input to the processing device 14 and controlled so that the pointer 18 is arranged at the reference position of the display screen 16a of the monitor 16. Thereby, the pointer 18 is arranged at the reference position of the display screen 16a. Then, the process returns to step 118.

  That is, in the present embodiment, when the user is facing the front, no matter how much the user moves the face and moves the position of the nose region (here, the center of gravity), the reference point 48 is Since it is updated with the movement, the display position of the pointer 18 does not move from the reference position of the display screen 16a. On the other hand, when the user moves the face in a direction different from the front direction and changes the direction of the face, the display position of the pointer 18 moves with the movement of the position of the nose region.

  In the present embodiment, whether or not the user's face is facing the front with respect to the camera 10 is determined as follows.

  As described above, the reference point update area 50 is determined from the reference point relative position information stored in the RAM 24 and the position of the face area 46. The center of gravity position of the current nose area detected by the tracking is the reference point update. If it exists outside the region 50 (see also the upper diagram in FIG. 6A), it is determined that the face is not facing the front, and the center of gravity position of the nose region exists within the reference point update region 50. If it is to be performed (see also the upper diagram in FIG. 6B), it is determined that the face is not facing the front.

  If the flowchart shown in FIG. 3 is followed, as shown in FIG. 6A, the recognition object 30 moves in a state where the face is not facing the front and each face part is detected (FIG. (See also 30a of 6 (A).) If the position of the nose region has changed, a negative determination is made in steps 118 and 120, so the relative position of the position 48a of the nose region after the change with respect to the latest reference point 48. Accordingly, the position of the pointer 18 moves.

  Also, as shown in FIG. 6B, when the position of the nose region changes with the face facing forward (when the entire face moves while facing the front of the camera 10). (See also B) 30b.) Since the determination at step 118 is affirmative, the nose region position 48b after the change is updated to the latest reference point 48, and the pointer 18 is displayed on the display screen 16a. It is displayed at the reference position. That is, the pointer 18 can remain stationary at the reference position.

  As described above, according to the present embodiment, when the predetermined face orientation is detected, the reference point 48 is updated, and the pointer 18 is moved to the predetermined reference position on the display screen 16a. Since it is configured to be controlled so as to be arranged, even if the user changes his / her posture, the user can easily put the image at the reference position on the display screen without pressing any reset button or the like with his hand. It can be arranged, and the movement of the image can be easily continued even after the posture change.

  In the present embodiment, the reference position of the pointer 18 is the center position of the display screen 16a. However, the present invention is not limited to this, and may be a position slightly deviated from the center, for example. 1 may be predetermined according to the application to which 1 is applied, or may be manually specified by the administrator, and is not particularly limited.

  In the present embodiment, an example in which the image processing memory 26 is a memory for temporarily storing image data representing a captured image captured by the camera 10 has been described. However, the present invention is not limited to this. is not. For example, the image processing memory 26 may be a memory used as a temporary storage area for storing information generated in the course of image processing. More specifically, for example, it is used as a memory for temporarily storing each information of hue, saturation, and luminance extracted when obtaining a predicted face detection area predicted to have a face from a captured image. Etc.

  In addition, since image data representing a captured image can be held in, for example, an LSI that constitutes a CPU, in the case of an algorithm that does not refer to past image data in a process such as face part detection processing, A configuration in which the image processing memory 26 is not provided is also possible.

  In the present embodiment, the state in which the user faces the front with respect to the camera 10 has been described as a “predetermined direction”, but the present invention is not limited to this, and a direction different from the front direction. It is good.

  The monitor 16 that displays the pointer 18 may be a monitor 16 connected to a personal computer or the like, or may be a large monitor 16 that is installed on the street for advertising purposes and can be viewed by passers-by. Good.

  In the latter case, for example, as shown in FIG. 7, the display screen 16 a of the monitor 16 is composed of a plurality of screen areas 17, and the reference position is individually set in each of the plurality of screen areas 17. You may be allowed to. In this case, when the pointer 18 is arranged at the reference position, which reference position is arranged may be determined as follows, for example.

  A table in which the reference position information and the position information indicating the position of the face area are registered in advance in the ROM 22 in correspondence with each of the plurality of pieces of identification information indicating the plurality of screen areas.

  Then, among the plurality of screen areas, a screen area corresponding to position information indicating the position of the user's face area 46 when the orientation of the face facing the camera 10 is detected. Information is extracted by referring to the table, and information indicating the reference position is input to the processing device 14 so that the pointer 18 is arranged at the reference position stored in the table corresponding to the extracted screen area. The display may be controlled.

  Accordingly, for example, if the face area position and the screen area position are registered in association with each other in the above table so that the position of the face area is substantially the same, the face area 46 of the user is detected by the face detection. When it is detected that the user is present in the region near the right end of the captured image of the camera 10, when it is detected that the user is facing the front, the screen arranged at the right end among the plurality of screen regions. The pointer 18 can be controlled to be arranged at the reference position of the area. Thereby, it can suppress that the pointer 18 is arrange | positioned at the reference | standard position of a screen area | region far from a user's face, and it can prevent that a user loses sight of the pointer 18. FIG.

  In the above embodiment, the face part area 44 used when operating the pointer 18 and the face part area 44 that is the basis of the reference point update area 50 used for detecting the face orientation are both noses. Although the case of the area has been described, the present invention is not limited to this. For example, the face part area 44 used when operating the pointer 18 may be different from the face part area 44 that is the basis of the reference point update area 50 used for detecting the face orientation. For example, the face part area 44 used when operating the pointer 18 is a nose area, the face part area 44 that is the basis of the reference point update area 50 used for detecting the face orientation is a mouth area, etc. It is. In this case, for example, in step 114, information on the relative position of the center of gravity position of the detected mouth area with respect to the face area 46 (hereinafter referred to as face part relative position information) is stored and held in the RAM 24, and the reference point update area 50 may be a square area having a predetermined size with a position determined from a relative position indicated by the face part relative position information with respect to the latest detected face area 46 as a center of gravity position. In step 118, it may be determined whether or not the user's face is facing the front depending on whether or not the detected center of gravity of the latest mouth area is outside the reference point update area 50. On the other hand, the reference point 48 used for the movement of the pointer 18 can be controlled as the center of gravity of the nose region in the same manner as in the above embodiment.

  Furthermore, the reference point update area 50 used for detecting the orientation of the face has a predetermined size in which the center point is the midpoint between the positions (for example, centroid positions) of the two different face part areas 44. It is good also as an area.

  Further, the reference point update region 50 does not have to be square, but may be rectangular or circular.

  Furthermore, the method for detecting the orientation of the face is not limited to the example described above. For example, a technique described in JP2003-141551A may be used. Briefly describing this method, image data obtained by photographing a human face with a camera is input, and the image data is compared with a plurality of face orientation templates in which information on the face orientation is stored in advance. A face orientation calculation method for calculating the orientation of the face photographed in the image data or extracting the feature points of the image data and calculating the face orientation from the positional relationship of the feature points. One rotation axis in the rotation of the three-dimensional space is set as the central axis of the body, the rotation angle of rotation that is rotation around this central axis is set as ψ, and the rotation angles around the other two rotation axes are set as θ and φ. In this case, the orientation of the face is output at the rotation angles ψ, θ, and φ, and the most frequent face orientation is defined as the front orientation from the input image data, and the defined front orientation is defined. For a face of orientation, three rotation angles ψ, θ, φ 0 is output for all, and the rotation angle ψ is output for the face rotated only around the central axis from the input image data, and the other two rotation angles θ and φ are output. A value close to 0 is output. Thereby, if all the rotation angles are 0, it can be determined that the user is facing the front, and otherwise, it can be determined that the face is not facing the front.

  In the above embodiment, the camera 10 and the image processing device 12 are described as independent devices. However, the image processing system 1 may be configured as a module in which the camera 10 and the image processing device 12 are integrated. Further, the module may be provided with a connection interface common to the processing apparatus 14 and configured to be detachable from the processing apparatus 14 via the connection interface. More specifically, for example, each of the module and the processing device 14 includes, for example, a USB (Universal Serial Bus) interface, and is connected via the USB interface, thereby imaging a user's face and operating a pointer. You may comprise so that execution of a function is enabled.

  The image processing system 1 according to the present embodiment can be applied to various applications. For example, the image processing system 1 may be applied to an application that displays advertisements, signs, and the like on the above-described large street monitor. At that time, for example, when it is detected that the user has approached the monitor, the face detection processing program shown in FIG. 3 is started and control is performed so that the pointer 18 appears on the display screen 16a. The user moves the pointer 18 by moving the face as described above. Further, for example, the present invention can be applied to an application or the like that designates the type of advertisement displayed on the display screen 16a by moving the pointer 18 and switches the advertisement to be displayed.

  According to such a configuration, for example, a user whose hands are occupied by shopping or the like can switch the display by operating the pointer 18 by moving his / her face, so that a desired operation can be performed without using a hand. The advertisement can be browsed, and the operability is greatly improved.

  Furthermore, the image processing system 1 of the present embodiment is useful even when a person with a handicap operates the personal computer. Even a handicapped person can easily move a personal computer or the like because the pointer 18 can be moved as long as the face is moved.

  Further, the image processing system 1 can be applied not only to a designation operation for designating a coordinate position by moving the pointer 18 but also to a gesture operation, for example. For example, when a circle is drawn once with the pointer 18, the user moves the face and moves the pointer like a double click when the circle is drawn twice with the pointer 18. This is an operation of inputting some command by moving 18 and drawing a predetermined trajectory. Thus, the image processing system 1 can be applied to various applications.

DESCRIPTION OF SYMBOLS 1 Image processing system 10 Camera 12 Image processing apparatus 14 Processing apparatus 16 Monitor 16a Display screen 17 Screen area 18 Pointer 20 CPU
22 ROM
24 RAM
26 Image processing memory 30 Recognition object 40 Predicted face detection area 42 Predicted face part area 44 Face part area 46 Face area 48 Reference point 50 Reference point update area

Claims (5)

Feature detection means for detecting a feature of the user's face from a captured image of the user;
Orientation detection means for detecting the orientation of the user's face from the captured image;
Each time a predetermined face orientation is detected by the orientation detection means, the feature position detected by the feature detection means when the predetermined face orientation is detected is the latest reference. Storage processing means for storing in the storage means identifiable as a point;
The predetermined image displayed on the display screen is moved based on the relative position of the feature part detected by the feature part detection unit with the latest reference point stored in the storage unit as a reference. First control means for controlling display of an image;
Second control means for controlling display of the image so that the image is arranged at a predetermined reference position on the display screen when the predetermined face orientation is detected by the orientation detection means. When,
An image processing apparatus. The display screen is configured by a plurality of predetermined screen areas, and the reference position is individually set for each of the plurality of screen areas,
The second control unit is configured to select a screen region corresponding to a position of the user's face when the predetermined face direction is detected by the direction detection unit among the plurality of screen regions. The image processing apparatus according to claim 1, wherein display of the image is controlled so that the image is arranged at a set reference position. Imaging means for imaging a user;
A feature detection unit for detecting a feature of the user's face from a captured image captured by the imaging unit;
Orientation detection means for detecting the orientation of the user's face from the captured image;
Each time a predetermined face orientation is detected by the orientation detection means, the feature position detected by the feature detection means when the predetermined face orientation is detected is the latest reference. Storage processing means for storing in the storage means identifiable as a point;
The predetermined image displayed on the display screen is moved based on the relative position of the feature part detected by the feature part detection unit with the latest reference point stored in the storage unit as a reference. First control means for controlling display of an image;
Second control means for controlling display of the image so that the image is arranged at a predetermined reference position on the display screen when the predetermined face orientation is detected by the orientation detection means. When,
An image processing system. A feature of the user's face is detected from a captured image of the user, and the orientation of the user's face is detected,
Each time a predetermined face orientation of the user is detected from the captured image, the position of the feature detected when the predetermined face orientation is detected is identified as the latest reference point. Memorize it in the memory means,
Based on the relative position of the feature portion relative to the latest reference point stored in the storage means, the display of the image is controlled so that a predetermined image displayed on the display screen moves,
Controlling the display of the image so that the image is arranged at a predetermined reference position on the display screen when the predetermined face orientation is detected;
Image processing method. Computer
Feature detecting means for detecting a feature of the user's face from a captured image of the user;
Orientation detection means for detecting the orientation of the user's face from the captured image;
Each time a predetermined face orientation is detected by the orientation detection means, the feature position detected by the feature detection means when the predetermined face orientation is detected is the latest reference. Storage processing means for storing in the storage means identifiable as a point,
The predetermined image displayed on the display screen is moved based on the relative position of the feature part detected by the feature part detection unit with the latest reference point stored in the storage unit as a reference. A first control unit for controlling display of an image; and when the predetermined face orientation is detected by the direction detection unit, the image is arranged at a predetermined reference position on the display screen. Second control means for controlling display of the image;
Program to function as.

Images