JP4781181B2 - User interface program, apparatus and method, information processing system - Google Patents

Description

  The present invention relates to user interface technology.

2. Description of the Related Art Conventionally, it is known that a user's operation is photographed using an imaging device such as a video camera, and a user's moving image is displayed on a screen so that a command can be input or a game can be played (for example, patents). Reference 1). In such an image processing apparatus, a user can input a command when a user's moving image contacts a menu screen or an object arranged on the screen. That is, the user's moving image itself functions as an input interface.
Japanese Patent Laid-Open No. 2005-216061

  In an application such as a game that uses a user's moving image as an input interface as described above, it is important to derive the behavior of the user who operates the application in a natural manner by means of effects on the screen. If the user requests an unnatural movement that is not in daily operation, the user's interest in the application may be lost. Further, in a state where no instruction is displayed on the screen, the user does not know what the user can perform a desired operation, and the application may not be started or continued.

  The present invention has been made in view of these problems, and an object of the present invention is to provide a technique for realizing an interface that is natural and easy for a user to use in an application that uses a user's moving image as an input interface. .

  An aspect of the present invention detects a user's face motion using a region specifying function for specifying a user's face region from the user's image captured by an imaging device, and the specified face region image. An event execution function for holding a motion detection function and a table associating a face motion with an event, searching the table for an event corresponding to the face motion detected by the motion detection function, and executing the event; This is a user interface program to be exhibited on a computer.

According to this aspect, it is possible to provide an input to the computer with a very simple and light operation such as the movement of the face, and it is possible to provide an input interface that is natural and easy to use for the user.
Note that the captured user image may be displayed on the display screen as it is, or a mirror image of the user may be displayed. The user image may not be displayed on the screen. An image of a character that moves the same as the user's movement may be displayed on the screen. Moreover, since the image of the face area may be specified, the user may be wearing a mask.

  The “event” includes an operation instruction corresponding to a button operation of a game controller, a TV remote control, etc., and an object screen display. The operation instruction includes help display, cancellation, determination, selection, menu display, termination, input of the movement direction, and the like.

  The motion detection function may detect the tilt of the user's face as a motion. The motion detection function determines which axis of the detected three-dimensional coordinates the user's face tilt is detected on, and the event execution function is determined by the motion detection function. Different events may be executed according to the tilt axis of the face. By determining whether the rotation axis of the face is about x, y, or z, three different events can be executed only by the inclination of the face. Different events may be executed depending on the rotation direction with respect to the axis, that is, clockwise or counterclockwise.

  The area specifying function further specifies a part area constituting the face within the specified face area, and the face action detected by the action detecting function includes a part action. The facial parts include eyes, mouth, nose, ears, eyebrows, pupils, tongues, etc., but are not limited to these as long as they can be identified by face recognition technology. According to this, various inputs can be given to the computer simply by changing the facial expression. In addition, since it does not involve the movement of the whole body, it is possible to realize an operation while sitting on a chair or lying down, which was difficult with this type of conventional application.

  The event executed by the event execution function may include an operation instruction for an application executed on the computer. In this case, it is preferable that at least a part of the face motion associated with the operation instruction in the table is an operation that approximates the operation when the user expresses the operation instruction in the real world.

  The motion detection function may count the number of repetitions of the user's face motion, and the event execution function may execute different events depending on the number of repetitions. The motion detection function may measure a time during which the face motion is continued, and may determine that the face motion has been executed when the measured time is equal to or greater than a threshold value. The face motion is often executed unconsciously by the user, but the occurrence of malfunction can be suppressed by including conditions such as the number of repetitions and duration.

  The event execution function may include an object generation function that generates an object when the event associated with the face motion detected by the motion detection function is a screen display of the object. The object generation function may change the display form of the object as time elapses or the inclination angle of the face increases or decreases. The display form here includes changing the color and size of an object, changing the speed of movement and rotation, and the like. By changing the object according to the user's movement, the user can be guided to perform the assumed input movement.

  It is also effective as an aspect of the present invention to replace the components and expressions of the present invention among methods, systems, computer programs, recording media storing computer programs, and the like.

  According to the present invention, it is possible to provide an interface that is natural and easy for a user to use in an application that uses a user's moving image as an input interface.

Embodiment 1 FIG.
FIG. 1 shows a configuration of an image processing system 100 using an image processing apparatus 10 according to an embodiment of the present invention. The image processing system 100 includes a camera 2 that is an imaging device that captures a moving image of a user, an image processing device 10, and a display device 4 that displays an image.

  The image processing device 10 continuously captures a moving image obtained by photographing the user 6 facing the display device 4 with the camera 2 in time series to generate a mirror moving image, and the mirror moving image 8 is displayed on the display device 4. To display in real time. The image processing apparatus 10 can also synthesize object images such as menus and cursors in addition to the mirrored moving image 8 and display them on the display device 4. The user 6 operates an application such as a game or album display executed by the image processing apparatus 10 while viewing the mirrored moving image and the object. When the user performs a predetermined operation, an image reflecting the user's operation is displayed on the screen, and some instruction is given to the application.

  The mirror moving image is generated by performing mirror processing (image left-right reversal processing) on the moving image captured from the camera 2 by the image processing apparatus 10. By this mirror surface processing, the user can operate the application as if looking at the mirror. The mirror surface process may be performed by a function of the camera 2.

  The camera 2 is preferably a CCD camera, but may be other analog or digital video cameras. As shown in the figure, the camera 2 is preferably installed on top of the display device 4. This is because the user can operate the application while looking at his / her appearance by photographing the user's operation facing the display device 4.

  FIG. 2 is a hardware configuration diagram of the image processing apparatus 10. The image processing apparatus 10 includes a CPU 12, a main memory 14, a GPU 16, an input / output control device 18, a drive device 20, a display control device 24, and a DMAC 26. These are connected to each other via a bus 28.

  The CPU 12 operates the operating system to control the entire image processing apparatus 10 and reads programs and data from the recording medium 50 mounted on the drive apparatus 20 to the main memory 14 and executes various processes according to the programs and data. . Further, the CPU 12 performs geometry processing for expressing the shape, movement, etc. of the object on the three-dimensional object data read from the recording medium 50 and passes it to the GPU 16.

  The GPU 16 holds drawing data, reads necessary drawing data in accordance with a command from the CPU 12, performs rendering processing, and draws it in a frame memory (not shown). The DMAC 26 controls DMA transfer for each circuit connected to the bus 28.

  The drive device 20 is a drive device that uses a removable recording medium 50 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. The image processing apparatus 10 functions as a user interface device by reading from the drive device 20 a recording medium 50 that stores a user interface program having the functions described below.

  An image signal from the camera 2 is input to the input / output control device 18. The input / output control device 18 is connected to an external storage device such as a hard disk drive, a communication device such as a modem or terminal adapter (TA) for communicating with an external device via a network, a TV tuner, a printer, and the like. Also good. A controller 52 for inputting data to the image processing apparatus 10 may be connected.

The display device 4 is connected to the display control device 24. The display control device 24 generates a video signal for enabling the data drawn in the frame memory by the GPU 16 to be displayed on the display device 4.
Note that the user interface program may be incorporated in advance in the image processing apparatus 10 instead of being loaded from the recording medium as described above.

  FIG. 3 shows the configuration of the user interface device according to the present embodiment. These configurations are realized by a CPU, a memory, a program loaded in the memory, and the like, but here, functional blocks realized by their cooperation are depicted. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The image input unit 102 captures a moving image captured by the camera 2 frame by frame. If the input image is an analog image, the image is captured after A / D conversion is performed to convert it to a digital image.

  The image reversing unit 104 performs mirror surface processing on the image captured by the image input unit 102 to generate a mirror moving image.

  The area specifying unit 106 specifies an area corresponding to the user's face for each frame image constituting the specular moving image. The area specifying unit 106 includes a face / part detection unit 122, a hand detection unit 124, and a reference image storage unit 126.

  The face / part detection unit 122 extracts the reference image of the face stored in the reference image storage unit 126, executes matching between the reference image and the input image, and obtains an image of an area corresponding to the user's face. Identify. When the face area is specified, parts further constituting the face, for example, areas corresponding to eyes, nose, and mouth are specified by matching with the reference image of each part.

In the reference image storage unit 126, reference images for front, sideways, upward, and downward are prepared. The face / part detection unit 122 performs matching between each of these reference images and the input image, and detects the face with the best match so that the face of the user can face in any direction. Can be specified. In the front face state, the angle of the face can also be detected by executing the matching between the input image and the tilted reference image using the neutral position when both eyes are parallel.
The hand detection unit 124 detects a motion of a user's hand by performing motion detection at a predetermined portion in the screen. For this detection, any known technique can be used.

  The motion detection unit 108 refers to a plurality of frames of the face region image and various part region images specified by the region specifying unit 106 to detect the user's face operation and part operation. The detailed configuration of the operation detection unit 108 will be described later with reference to FIG.

  The event execution unit 110 holds a table that associates the face motion and part motion with the event, searches the table for an event corresponding to the face or part motion detected by the motion detection unit 108, and searches for the event. Execute. “Event” includes an operation instruction for the application and a screen display of the object. The operation instruction includes help display, cancellation, determination, selection, menu display, termination, input of the movement direction, and the like. The detailed configuration of the event execution unit 110 will be described later with reference to FIG.

  The image synthesizing unit 116 draws a synthesized image obtained by synthesizing the specular moving image generated by the image inverting unit 104 and the object image generated by the event executing unit 110 in a frame memory (not shown). The display control unit 118 controls display of the image synthesized by the image synthesis unit 116 on the display device 4.

  In relation to the hardware configuration shown in FIG. 2, the image input unit 102 includes the input / output control device 18 and the like. The image reversing unit 104, the region specifying unit 106, the operation detecting unit 108, and the event executing unit 110 are configured by the CPU 12, the main memory 14, the GPU 16, and the like. The image composition unit 116 is configured by the GPU 16, and the display control unit 118 is configured by the GPU 16 and the display control device 24.

  FIG. 4 is a functional block diagram illustrating detailed configurations of the operation detection unit 108 and the event execution unit 110. These functional blocks can also be realized in various forms by hardware only, software only, or a combination thereof.

The motion detection unit 108 includes a face tilt axis determination unit 150, a displacement amount determination unit 152, a part motion determination unit 154, and a screen state acquisition unit 156.
The face inclination axis determination unit 150 determines which axis of the three-dimensional coordinates defined on the screen the user's face inclination is about. As described above, the area specifying unit 106 holds reference images for front, sideways, upside, and downside faces, and specifies the face in which direction when specifying the face area. be able to. Therefore, the face tilt axis determination unit 150 acquires the face orientations specified by the region specifying unit 106 for a plurality of frames, and determines the face tilt axis according to the change in the face orientation during this period. A specific example of the face tilt axis will be described later. Changes in the orientation of the face can also be determined by changes in the number of eyes. For example, if both eyes are shown in a certain frame but one eye is seen in a later frame, it can be determined that the face is shaken sideways.

  After the face tilt axis determination unit 150 determines the face tilt axis, the displacement amount determination unit 152 calculates the tilted angle of the user's face in a plane perpendicular to the axis. In the area specifying unit 106, when the face in the input image is tilted, the face area is detected by matching the tilted reference image. Therefore, if the tilt angle of the reference image is calculated, the face tilt angle is calculated. Can be requested. The displacement amount determination unit 152 may determine that the face is tilted when the face inclination angle reaches a predetermined threshold, for example, 40 degrees or more. Further, the displacement amount determination unit 152 may measure a duration time during which the user is tilting his / her face. This can be calculated depending on how many frames the reference image is tilted to identify the face region. The displacement amount determination unit 152 may determine that the face is tilted when the measured time is equal to or greater than a threshold value in advance. Furthermore, the displacement amount determination unit 152 may count the number of face tilt repetitions.

  The part motion determination unit 154 acquires an image of a region for each part of the face specified by the region specifying unit 106 for a plurality of frames, and detects an operation for each part based on a difference between the plurality of frames. The detection operation is, for example, opening / closing of eyes (blinking), opening / closing of mouth, and the like.

  The screen state acquisition unit 156 acquires the screen state displayed on the display device 4 when the user performs face movements or hand movements. The screen state here refers to the arrangement and movement of objects on the screen, and the position of the user's mirrored moving image within the screen. By acquiring this screen state, it is possible to implement control such as a touch determination described later, or an event executed by a face motion when a specific object is placed.

The event execution unit 110 includes an input condition determination unit 158, a table storage unit 160, an operation instruction unit 162, an object generation unit 112, and an object data storage unit 114.
The table storage unit 160 stores a table in which the face motion determined or detected by the face tilt axis determination unit 150, the displacement amount determination unit 152, and the part motion determination unit 154 is associated with an event. The face motion includes the face tilt axis, the face tilt angle, and the part motion. The event includes an operation instruction to the application executed by the image processing apparatus 10 such as determination and cancellation, and an object on the screen. Includes display. An example of combinations of face motions and events defined in the table will be described later with reference to FIG.

  The input condition determination unit 158 searches the table storage unit 160 for an event corresponding to the face motion determined or detected by the face tilt axis determination unit 150, the displacement amount determination unit 152, and the part motion determination unit 154, and further includes a table. It is determined whether or not the condition defined in is satisfied.

  The object data storage unit 114 stores object data to be displayed on the screen of the display device 4. The object data includes a menu image, a cursor image, an image for decorating a user action, an image for configuring a game executed on the image processing apparatus, and the like.

The object generation unit 112 takes in the object data from the object data storage unit 114 and generates an object image. When the input condition determination unit 158 determines that the conditions defined in the table are satisfied, the object generation unit 112 generates an object image that is also defined in the table. The object generation unit 112 changes the display form of the object with time, or changes the display form of the object as the face inclination angle calculated by the motion detection unit 108 increases or decreases.
When the input condition determination unit 158 determines that the conditions defined in the table are satisfied, the operation instruction unit 162 executes an operation associated with the table.

  The input condition determination unit 158 also performs contact determination between the hand and the object. If it is determined that the hand of the specular image has touched the object arranged on the screen, the operation instruction unit 162 and the object generation unit 112 are instructed to execute an event associated with the object.

FIG. 5 is a flowchart for realizing the input interface based on the face motion according to the present embodiment.
First, a user image is acquired by the image input unit 102, and the user's mirror moving image created by the image reversing unit 104 is displayed on the display device 4 (S10). Next, the area specifying unit 106 specifies the face area and the parts area of the user (S12). The motion detection unit 108 detects the motion of the face and the motion of the part using the image of the face region and the image of the part region for a plurality of frames, respectively (S14). The motion detection unit 108 further determines the tilt axis and displacement amount of the detected face motion (S16).

  The input condition determination unit 158 determines whether or not an event associated with the face motion or part motion determined in S16 is defined in the table in the table storage unit 160 (S18). If not defined (N in S18), this flow is terminated. If it is defined (Y in S18), it is determined whether the conditions such as the displacement amount, the duration, and the number of repetitions detected in S16 satisfy the conditions defined in the table (S20). If the condition is not satisfied (N in S20), this flow ends. If the condition is satisfied (Y in S20), the event execution unit 110 executes an event associated with the table (S22).

FIG. 6 is a flowchart showing the face or part detection method in S12 of FIG.
First, the area specifying unit 106 acquires an input image for one frame (S30). A filtering process is performed on the image to remove the noise and effectively extract the feature amount (S32). Subsequently, pattern matching for comparing the reference image of the face stored in the reference image storage unit 126 with the input image is executed (S34). The reference image of the face is an image created based on the images of the faces of several tens to several hundreds of people. If the similarity with the reference image is high, the portion in the image may be a face. Increases nature. Therefore, the area specifying unit 106 calculates the similarity with the reference image for a certain area of the input image (S36). In many cases, the position and size of the face are different between the reference image and the input image. Therefore, the input image may be offset, or the input image may be enlarged or reduced before calculating the similarity between the input image and the reference image. There are various methods for calculating the similarity, and any method can be used. Since this method is known, a detailed description is omitted.

  The region specifying unit 106 compares the degree of similarity with a predetermined threshold value, and determines whether or not the degree of similarity is greater than or equal to the threshold value (S38). If it is less than the threshold value (N in S38), it is determined that this region is not a face, so the process returns to S34 and the matching with the reference image is continued for other parts of the input image. If the similarity is equal to or greater than the threshold value in S38 (Y in S38), the face area is determined (S40).

When the face area is determined, a face part area, for example, an eye, nose, or mouth area is searched for in the face area image. A partial image in the face region is compared with the reference image of the part (S42), and the similarity is calculated in the same manner as described above (S44). The region specifying unit 106 compares the similarity with a predetermined threshold for each part, and determines whether the similarity is equal to or greater than the threshold (S46). If it is less than the threshold value (N in S46), the process returns to S42, and pattern matching with the reference image is continued for the other part of the face region image. If the degree of similarity is equal to or greater than the threshold value in S46 (Y in S46), the part area is determined (S48). Other methods may be used to detect the part region. For example, after determining the eye and mouth area, the nose position may be determined by estimating the position of the nose from the relative positional relationship between the eyes and the mouth.
As described above, the position and size of the user's face, the positions of parts such as eyes, nose and mouth can be detected from the input image.

  Since the detection of the face region by calculating the similarity is well known, further explanation is omitted. The face area may be specified by a method other than the above.

  7A to 7C are schematic diagrams illustrating examples of face tilt axes. (A)-(c) represents the specular image of the user projected on the display apparatus. Assume 3D coordinates in the space in the screen. As shown in (a), when the user moves to tilt his / her neck, it is determined that the inclination is about the x axis. As shown in (b), when the user moves upward or downward, it is determined that the inclination is about the y axis. As shown in (c), when the user moves to the left or right, it is determined that the inclination is around the z axis. As described above, the determination of the face tilt axis is performed based on the change in the face orientation. For example, if the face orientation in a certain frame is determined to be the front face and the face orientation after several frames is determined to be the left face, the face tilt axis can be determined to be the z-axis. The same applies to the x axis and the y axis.

  FIG. 8 shows an example of the table 40 stored in the table storage unit that associates facial motions with events. The “motion part” column 30 represents a part such as a face, eyes, or mouth where the motion is detected by the motion detection unit 108. The “motion content” column 32 indicates the motion detected by the motion detection unit 108, and indicates the direction of rotation if the motion part is a face, and opening and closing if the eye or mouth. “Hiding” the eye means that the eye image is not detected at a desired position in the face area because the eye is covered with a hand or the like. The “condition” column 34 is a condition for determining that the operation content shown in the operation content column 32 has been made, and is determined by the input condition determination unit 158. This condition includes the face inclination angle detected by the displacement amount determination unit 152, the duration of the motion, the number of repetitions of the motion, and the like.

The “object” column 36 represents an object to be displayed on the screen when the operations defined in the columns 30 to 34 in a certain row are satisfied. As shown in the figure, the object includes a character string such as “?” Mark, “OK”, “how to use guide”, a symbol such as “maru”, “X”, etc., as well as erasing the displayed object. The “operation instruction” column 38 represents an operation instruction to be issued to the application being executed in the image processing apparatus when the operations defined in the columns 30 to 34 in a certain row are satisfied. This operation instruction includes help display, cancellation, determination, selection, menu display, termination, input of the moving direction, and the like.
For example, referring to the second line from the top of the table 40, when the detected user's face is rotated 40 degrees around the x axis, the object generation unit 112 displays the object “?” On the screen, Further, the operation instruction unit 162 instructs the application to display a help menu.

  In addition, it is preferable that the face motion associated with the operation instruction in the table 40 is an operation that approximates the operation when the user expresses the operation instruction in the real world. For example, when there is something that humans do not understand, an operation of tilting the neck as shown in FIG. Therefore, this movement is determined as an inclination around the x-axis, and is associated with an event for displaying a help menu for solving the unknown. In addition, when consenting, an operation is performed as shown in FIG. 7B, and this movement is determined as a tilt around the y axis, and is associated with an event for issuing a determination operation instruction. Further, when negating, an operation of swinging the head sideways is performed as shown in FIG. 7C. This movement is determined as rotation around the z axis, and is associated with an event for issuing a cancel operation instruction. As a matter of course, the combination of the action and the event is not limited to this, and it may be associated with an event that is not related at all or an event having the opposite meaning to the action. If it is the latter, it can also improve game property by requesting the reverse operation from the user.

  In addition, since the meaning represented by human gestures may vary depending on the race and country of the user, a plurality of tables are prepared in advance, and when the user selects a country, a table suitable for that is used. Also good.

  Based on the configuration of the image processing system as described above, some examples of screen operations using the face interface according to the present embodiment will be described.

Example 1.
FIGS. 9A to 9C show examples of screen changes in the embodiment in which help appears when the head is tilted. When something is known while the application is in progress, when the user performs an operation of tilting the face around the x axis (FIG. 9A), the motion detection unit 108 detects the face area through The tilt axis and tilt angle of the face are determined. When the tilt angle of the face reaches the condition defined in the table, the object generation unit 112 displays the object “?” In the vicinity of the head of the user's mirror image on the screen (FIG. 9B). When the user touches the appearing “?” Object with his / her hand, the input condition determination unit 158 determines contact, and the object generation unit displays a “Usage Guide” character on the screen (FIG. 9 ( c)). After that, a help screen is displayed. This help screen may also be operable by operations such as facial parts and hands, or may be operable by a controller or the like.

  FIGS. 10A to 10C show examples of screen changes in the embodiment where the help disappears when the head is shaken. In the state where the “Usage Guide” is displayed on the screen (FIG. 10 (a)), the need for help is eliminated, and the user swings his / her head (FIG. 10 (b)). 108 determines the tilt axis and tilt angle of the face through detection of the face area. When the input condition determination unit 158 determines that the number of times the face has been shaken has reached the conditions defined in the table, the object generation unit 112 displays “how to use” displayed near the head of the user's specular image. The character string object “Guide” flies toward the left end of the screen, and the effect of disappearing from the screen is produced (FIG. 10C). The object may disappear from the screen when the user performs a hand-shaking operation on the character string object of the “Usage Guide”.

Since the operation shown in FIG. 9 and FIG. 10 may be unconsciously performed by the user, it is preferable that the operation is recognized by repeating the operation a plurality of times in order to prevent malfunction. Further, it may be associated with different events depending on the number of repetitions of the operation.
In addition to associating events with the rotation of the x, y, and z axes, different events may be assigned to the face direction from the front, for example, the movement of the face in the upper right, lower right, upper left, and lower left. Good.

Example 2
FIGS. 11A to 11D show examples of screen changes in the embodiment for determining the duration of the operation.
When the user performs an operation of tilting the face around the x-axis when something is unknown while the application is in progress (FIG. 11A), the motion detection unit 108 detects the face area. The tilt axis and tilt angle of the face are determined. When the input condition determination unit 158 determines that the face inclination angle is equal to or larger than the angle specified in the conditions defined in the table, the object generation unit 112 near the head of the user's mirror image on the screen A small object “?” Is displayed on the screen (FIG. 11B). If the state where the face is tilted is continued, the object gradually expands according to the time (FIG. 11C), and if it is determined that the state has continued for a predetermined time (for example, 3 seconds), the object is The object generation unit 112 disappears and a character string object of “how to use guide” is displayed near the head of the mirror image (FIG. 11D). After that, a help screen is displayed on which items can be searched by an operation by a controller or the like. Once the object “?” Appears, when the user returns the face to the original state, the object gradually becomes smaller and disappears from the screen.

  In order to prevent a malfunction, an element determined by an angle may be incorporated. For example, when the user starts to tilt the face around the x axis, a small object “?” Is displayed in the vicinity of the user's mirror image on the screen. As the tilt angle increases, the object also expands. When the tilt angle exceeds a predetermined size, the object disappears and a character string object of “how to use guide” is displayed near the head of the specular image. Once the object “?” Appears, if the user decreases the tilt of the face, the object is also reduced accordingly and disappears from the screen. The display position of the object may be shifted according to the tilt angle of the face. When another menu is displayed on the screen first, the object generation unit 112 may change the appearance position of the object so that the object does not overlap the menu. You may comprise so that a user can instruct | indicate the appearance position of an object via the controller which is not shown in figure.

  The position where the object “?” Is displayed may be changed according to the direction in which the user tilts his / her face. For example, if an object is displayed on an extension line of the center axis of the user's face, the object is displayed near the head even if the user tilts the face in either the right or left direction. You may make it change the display position of object "?" According to the speed of the operation | movement which a user inclines a face.

As described above, by adding the continuation and size elements of the operation to the determination, it is possible to prevent an operation instruction such as selection or determination from being performed by the user's accidental operation, and the reliability of the operation can be improved. In addition, by displaying an object when a predetermined motion is detected and changing the display mode of the object according to the subsequent motion, the user can visually determine whether his motion is correct or incorrect. Can be recognized. That is, it is possible to give a natural suggestion to the user whether to continue or cancel the current operation.
In combination with this, a sound effect unit (not shown) may output a predetermined sound effect in accordance with a change in the display mode of the object. For example, in the example of FIG. 11, if the volume of the sound effect is increased as the object gradually expands, the user can be audibly informed that the user's action is detected by the device. it can.

  In FIG. 11, the operation at the time of help is described as an example, but the same can be done in the case of determination or cancellation.

Example 3
All or part of a general game controller function may be replaced with a face motion. For example, the rotation direction of the face, that is, upward, downward, leftward, and rightward is equivalent to the input of up / down / left / right of the controller cross key. Further, among the facial parts, for example, the opening / closing of the right eye corresponds to the pressing of a multi-button (general function is “OK”) and the cross button of the left eye opening / closing (general function is “Cancel”). In this way, more complicated operation instructions can be expressed by facial movements.
In addition, a menu screen corresponding to the face motion may be displayed on the display device. For example, the menu screen may be displayed in the direction the face is facing. Objects representing selection, execution, cancellation, etc. may be displayed in the displayed menu screen so that they can be selected by the action of the face.

Embodiment 2. FIG.
In the first to third embodiments, the case where the mirror image of the user is displayed on the display device has been described. However, as long as the user's operation can be recognized, the mirror image need not be displayed.

  Hereinafter, an embodiment in the case where the mirror image of the user is not displayed will be described.

Example 4
Consider an application that performs a slideshow of digital photos stored on a recording medium. It is assumed that the user rotates his / her face around the z axis while a photograph is displayed on the display device. When the motion detection unit 108 detects this motion, the event execution unit 110 moves the currently displayed photo in the direction facing the face, and displays the next photo on the display device. Further, it is assumed that the user rotates the face around the y-axis, that is, performs an operation of “whipping” in a state where a plurality of photographs are displayed on the display device. When the motion detection unit 108 detects this motion, the event execution unit 110 selects a photo that is currently in focus from among a plurality of photos, and displays the enlarged photo on the entire screen of the display device. Further, when the opening / closing of the eyes is detected by the motion detection unit 108 while the photograph is displayed on the display device, the event execution unit 110 ends the slide show.

Example 5 FIG.
Assume a first-person viewpoint game, that is, a game in which the viewpoint of the user facing the display device is displayed on the screen. In such a game, when a fork road is displayed on the screen, the motion detection unit 108 detects the direction in which the user faces his face, and the event execution unit 110 creates an image when the road in that direction is selected. To display. Further, when the user looks down, an image looking down is displayed, and when the user looks up, the sky can be seen, so that the user can feel more in the virtual space of the game.
A mirror image of the user may be displayed at one corner of the screen. By doing so, it is possible to notify the user that the user's operation is recognized by the image processing apparatus.

The present invention has been described with respect to several embodiments. According to the present invention, it is possible to provide an input with a very simple and light load operation such as a face movement, and it is possible to provide a natural and simple input interface for the user.
Also, it is important in an interactive user interface to react immediately to some action and to specify in real time what state it is now. In the present invention, since the object is displayed on the screen in accordance with the user's operation, it can be transmitted that the user's operation is recognized by the image processing apparatus. In addition, by changing the display form of the object displayed on the screen according to the user's action, the user can visually recognize whether the current action leads to a desired function, A continuation or discontinuation can be induced naturally. Furthermore, the user can unconsciously acquire how to handle and play while interacting with the image.

  The present invention has been described above based on the embodiment. It is to be understood by those skilled in the art that these embodiments are exemplifications, and various modifications are possible for each component or combination of processes, and such modifications are also within the scope of the present invention.

  Arbitrary combinations of the constituent elements described in the embodiment, and a representation of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium, and the like are also effective as an aspect of the present invention. In addition, the method described as a flowchart in the present specification includes processing executed in parallel or individually in addition to processing executed in time series according to the order.

  The application program such as a game and the user interface program of the present invention may be provided by the same recording medium or may be provided by different recording media. The user interface program of the present invention may be incorporated in advance in a game device or the like.

In the embodiment, it has been described that the mirror image of the user captured by the camera is displayed as it is on the display device, but a character may be displayed instead of the user and moved according to the user's action. The character may be a humanoid, or it may be an animal or a non-living creature if the face and hand movements are known. Objects that simply correspond to the eyes, nose, and mouth may be reflected.
Even in this case, the face / part area identification and the motion detection based on the mirror image of the user are always executed, and an event associated with the detected motion is generated as in the above-described embodiment.

  Motion detection according to the present invention may be used as an aid to speech recognition. When the image processing device has a voice input function for recognizing the voice uttered by the user and inputting the corresponding character, it also takes into account the mouth movement when the user says “Yes” or “No”. It is possible to issue a decision or cancel operation instruction.

1 is a diagram illustrating a configuration of an image processing system using an image processing apparatus according to an embodiment of the present invention. It is a hardware block diagram of an image processing apparatus. It is a figure which shows the structure of the user interface apparatus concerning this embodiment. It is a functional block diagram which shows the detailed structure of an operation | movement detection part and an event execution part. It is a flowchart which implement | achieves the input interface by the operation | movement of the face concerning this embodiment. It is a flowchart which shows the detection method of the face or parts in S12 of FIG. 7A to 7C are schematic diagrams illustrating examples of face tilt axes. It is a figure which shows an example of the table which linked | related the operation | movement of the face and the event memorize | stored in a table memory | storage part. FIGS. 9A to 9C are diagrams illustrating an example of a screen change in the embodiment in which help appears when the head is tilted. FIGS. 10A to 10C are diagrams illustrating an example of a screen change in the embodiment in which the help disappears when the head is shaken. FIGS. 11A to 11C are diagrams showing an example of a screen change in the embodiment for determining the duration of the operation.

Explanation of symbols

  2 camera, 4 display device, 10 image processing device, 40 table, 102 image input unit, 104 image inversion unit, 106 area specifying unit, 108 motion detection unit, 110 event execution unit, 112 object generation unit, 114 object data storage unit , 116 Image composition unit, 118 Display control unit, 120 Image comparison unit, 122 Face / part detection unit, 124 Hand detection unit, 126 Reference image storage unit, 150 Face tilt axis determination unit, 152 Displacement amount determination unit, 154 Parts operation Determination unit, 156 screen state acquisition unit, 158 input condition determination unit, 160 table storage unit, 162 operation instruction unit.

Claims (17)

An area specifying function for specifying an area of the user's face from the user's image captured by the imaging device;
An action detection function for detecting the tilt of the user's face as one of the face actions using an image of the identified face area;
An event execution function for holding a table associating a face motion and an event, searching the table for an event corresponding to the face motion detected by the motion detection function, and executing the event;
In allowed to demonstrate on the computer Ruyu over The interface program,
The event execution function has an object generation function for displaying a first object on the screen as an event corresponding to the inclination of the face detected by the motion detection function,
The object generation function changes the display size of the first object as the face inclination angle increases or decreases, and changes the first object display size when the face inclination angle exceeds a predetermined size. A user interface program that displays a second object different from an object on a screen. The motion detection function determines which axis of the three-dimensional coordinates defined on the screen the detected user's face inclination is,
The user interface program according to claim 1 , wherein the event execution function executes different events according to a tilt axis of the face determined by the motion detection function. The area specifying function further specifies an area of parts constituting the face within the specified face area,
The user interface program according to claim 1, wherein the motion of the face detected by the motion detection function includes the motion of the part.   The user interface program according to claim 1, further comprising a display function for displaying a menu screen on the screen according to the action of the face. 5. The user interface program according to claim 4 , wherein the event execution function can specify any one of selection, execution, and cancellation of a menu screen in accordance with a face motion. The event executed by the event execution function includes an operation instruction for an application executed on the computer, and at least a part of the facial motion associated with the operation instruction in the table is operated by the user. the user interface program according to any one of claims 1 to 3, wherein the instruction to an operation of operating an approximation of when expressed in the real world. The motion detection function counts the number of repetitions of the user's face motion,
The event execution function, the user interface program according to any one of claims 1 to 3, characterized in that to perform different event according to the number of repetitions. The motion detection function counts the number of repetitions of the user's face motion,
The event execution function, when the number of repetitions reaches a threshold, according to any one of 3 claims 1, characterized in that executing the associated wear event in the operation of the face in the table User interface program. The area specifying function holds a reference image for front, sideways, upward, and downward faces, detects the face orientation by matching the reference image and the image of the specified face area,
The user interface program according to claim 2 , wherein the motion detection function determines the tilt axis of the face according to a change in face orientation at a plurality of time points. The motion detection function detects the motion of the user's mouth,
When the user interface program further includes a voice input function for recognizing a voice uttered by the user and inputting a corresponding character, refer to the movement of the mouth detected by the motion detection function when the voice is recognized. The user interface program according to claim 3 , wherein: The motion detection function measures a time during which the facial motion continues, and determines that the facial motion has been executed when the measured time exceeds a threshold value. Item 11. A user interface program according to any one of Items 1 to 10 . Prior Symbol object creation function, a user interface program according to the display form of the object to claim 11, characterized in that to change over the time. 13. The user interface program according to claim 12 , further causing the computer to exhibit an image composition function for composing the mirror image of the user imaged by the imaging device and the object to generate one image. Recording medium characterized by storing a program according to any one of claims 1 to 13. An area specifying unit for specifying an area of the user's face from the image of the user imaged by the imaging device;
An action detection unit that detects the tilt of the user's face as one of the face actions using an image of the identified face area;
An event execution unit that holds a table associating a face motion and an event, searches the table for an event corresponding to the face motion detected by the motion detection unit, and executes the event ;
The event execution unit includes an object generation unit that displays a first object on the screen as an event corresponding to the inclination of the face detected by the motion detection unit,
The object generation unit changes the display size of the first object to be larger or smaller as the angle of face inclination increases or decreases, and when the angle of face inclination exceeds a predetermined value, A user interface device that displays a second object different from an object on a screen . An area of the user's face is identified from the user's image captured by the imaging device,
Using the image of the identified face area, the user's face tilt is detected as one of the face movements,
The first object is displayed on the screen as an event corresponding to the detected face inclination,
As the face tilt angle increases or decreases, the display size of the first object is changed to be larger or smaller, and when the face tilt angle exceeds a predetermined size, the second object is different from the first object. A user interface method characterized in that an object is displayed on a screen . Imaging means for imaging a user;
Display means for displaying a captured user image;
Area specifying means for specifying an area of the user's face from the user's image;
Motion detection means for detecting the tilt of the user's face as one of the face motions using an image of the identified facial area;
An event execution means for holding a table associating a face motion with an event, searching the table for an event corresponding to the face motion detected by the motion detection means, and executing the event ;
The event execution means includes object generation means for displaying a first object on the screen as an event corresponding to the inclination of the face detected by the motion detection means,
The object generation means changes the display size of the first object as the face inclination angle increases or decreases, and increases or decreases the display size of the first object. An information processing system that displays a second object different from an object on a screen .

Images