JP2001306049A - Information processor, movement recognition process displaying method and program storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make a user learn how movements of hands of the user are to be recognized. SOLUTION: This invention can make a user learn how movements of hands of the user are recognized by recognizing movements in images, based on images obtained by picking up images of the hands of the user with a CCD camera 8 and by generating visual feedback pictures expressing loci of the recognized movements of the hands of the user and by displaying the pictures on a liquid crystal display 10.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing apparatus, an operation recognition process display method, and a program storage medium, which are suitably applied to, for example, a notebook personal computer (hereinafter, referred to as a notebook personal computer). .

[0002]

2. Description of the Related Art Conventionally, a notebook personal computer is constituted by display means such as a liquid crystal display and input means such as a keyboard or a mouse for inputting predetermined commands and characters, and responds to commands input by key operations. To execute a predetermined process, and display the execution result on a display unit.

In recent notebook personal computers, a so-called jog dial having a predetermined shape is used as input means other than a keyboard and a mouse, for example, so as to slightly protrude from the side of the housing of the notebook personal computer. In response to a rotating operation and a pressing operation on the jog dial, a command such as selection of a menu item and determination of a command is input.

[0004]

By the way, in a notebook personal computer having such a configuration, in addition to the above-mentioned keyboard, mouse or jog dial, a user is imaged through an externally connected camera, and the image is taken in accordance with the movement of the user. It has been suggested that commands be entered automatically,
When a user inputs a wrong command, the user cannot know the operation based on which the command is recognized, and there is a problem that the usability is poor.

The present invention has been made in view of the above points, and an information processing apparatus capable of feeding back and learning a recognition process until a motion of a recognition target is recognized for a user, and a motion recognition process display method. And a program storage medium.

[0006]

According to the present invention, a moving direction of a recognition target in an image is recognized based on an image obtained by imaging the recognition target by an imaging unit. By generating a recognition process image representing the trajectory of the moving direction of the recognized recognition target and displaying the image on a predetermined display unit, the user is fed back to the user how the moving direction of the recognition target has been recognized. Can be learned.

[0007]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below in detail with reference to the drawings.

(1) First Embodiment (1-1) External Configuration of Notebook-Type Personal Computer In FIG. 1, reference numeral 1 denotes a notebook-type personal computer (hereinafter referred to as an information processing apparatus) to which the present invention is applied as a whole. , This is called a notebook computer).
And a display unit 3 attached to the main body 2 so as to be openable and closable.

The main body 2 has a plurality of operation keys 4 for inputting various characters, symbols, numbers, and the like on a top surface thereof, and a stick type pointing device (hereinafter, simply referred to as a stick) 5 used for moving a mouse cursor. Left click buttons 5A and 5B corresponding to a left button and a right button of a normal mouse, a center button 5C for operating a scroll bar without moving the mouse cursor to the scroll buttons, built-in speakers 6A and 6B,
Push-type power switch 7, CCD provided on display unit 3
(Charge Coupled Device) A shutter button 9 for a camera 8, a power lamp PL, a battery lamp BL, and a message lamp M each composed of an LED (Light Emitting Diode).
L and the like are provided.

The display section 3 has a TFT (Thin Fil) corresponding to, for example, an 8.9 type (1024 × 480 pixels) on its front surface.
m Transisitor) Liquid crystal display 1 consisting of color liquid crystal
0 is provided, and an image pickup section 11 provided with a CCD camera 8 as an image pickup means is provided at the upper end of the front center so as to be rotatable with respect to the display section 3.

In the image pickup section 11, the CCD camera 8 can be rotated at an angle of about 180 degrees from the front direction to the back direction of the display section 3 and positioned at an arbitrary angle. Focus adjustment when imaging a desired imaging target can be easily performed by rotating the adjustment ring 12 provided at the upper end of the imaging unit 11.

The display unit 3 is provided with microphones 13 on the front side and the back side near the left end of the image pickup unit 11, and covers a wide area from the front side to the back side of the display unit 3 via the microphone 13. It is designed to be able to collect sound across.

The display unit 3 further includes a liquid crystal display 10
The pawls 14 and 15 are provided near the left end and the right end, respectively, and holes 16 and 17 are provided at predetermined positions of the main body 2 corresponding to the pawls 14 and 15, respectively. In this state, the claws 14 and 15 are fitted into the corresponding holes 16 and 17, respectively.

On the other hand, when the front side of the display unit 3 closed by the main body 2 is lifted, the fitted state of the holes 16 and 17 and the claws 14 and 15 is released, and As a result, the display unit 3 can be developed from the main body 2.

The body 2 has an IrDA (In) on its right side.
frared Data Association) compliant infrared port 18,
Headphone terminal 19, microphone input terminal 2
0, USB (Universal Serial Bus) terminal 21, external power supply connector 22, external display output connector 2
3. A jog dial 24 and a modem terminal 25 for a modular jack are provided for inputting a command for executing a predetermined process by a rotation operation and a pressing operation of the rotary operation element.

On the other hand, as shown in FIG. 2, the main body 2 has an exhaust hole 26 and a PCMCIA (Personal Computer) on its left side.
Memory Card International Association) standard PC
(Personal Computer) card compatible with PC card slot 27 and 4-pin IEEE (Institute of E)
Electrical and Electronics Engineers) 1394 terminal 28 is provided.

Further, as shown in FIG. 3, the main body 2 is provided with a battery connector 29 on a rear side surface thereof, and a slide type removal lever 31 for detaching the battery pack 30 (FIG. 1) on the bottom surface, and the slide type removal lever 31. Lock lever 32 for locking the sliding operation of lever 31
And a reset switch 33 for interrupting the operation of the main body 2 and reconstructing the environment at power-on. The battery pack 30 is detachably connected to the battery connector 29.

(1-2) Circuit Configuration of Notebook Type Personal Computer Next, the circuit configuration of the notebook personal computer 1 will be described in detail with reference to FIG. In the main body 2 of the notebook personal computer 1, a CPU for controlling various functions of the main body 2 in an integrated manner
(Central Processing Unit) 50 is connected to the host bus 52, and the CPU 50 controls the RAM (Rand Rand).
om Access Memory) 53 is executed at a predetermined operating speed based on a system clock provided from the clock generator 60, thereby realizing various functions. I have.

A cache memory 51 is connected to the host bus 52 so as to cache data used by the CPU 50 and realize high-speed access.

The host bus 52 is connected to a PCI (Peripher
al Component Interconnect) bus 55 and host-PC
The PCI bus 55 is connected via a video controller 56 and an IEEE 1349
Interface 57, video capture processing chip 8
3 and the PC card interface 58 are connected.

Here, the host-PCI bridge 54
It controls the exchange of various data between the PU 50 and the video controller 56, the video capture processing chip 83, the IEEE 1349 interface 57, and the PC card interface 58, and controls the memory bus 5
9 for controlling the memory of the RAM 53 connected thereto.

The host-PCI bridge 54 includes a video controller 56 and an AGP (Accelerated Graphics P
(ort), so that image data can be transferred between the host-PCI bridge 54 and the video controller 56 at a high speed.

The video capture processing chip 83 includes an I ² C bus 82 (generally an SM (System
Management) bus, and when image data captured by the CCD camera 8 is supplied via the I ² C bus 82, the image data is stored in a built-in frame memory (not shown). And store it in JPEG (Joint
Photographic Experts Group) After generating JPEG image data by performing image compression processing according to the standard,
The JPEG image data is stored in the frame memory again.

Then, the video capture processing chip 83
Transmits the JPEG image data stored in the frame memory to the RAM 53 using the bus master function in response to a request from the CPU 50, and then transfers the JPEG image (still image) data or the Motion JPEG image (moving image) data to the hard disk drive ( HDD) 67.

The video controller 56 outputs image data based on various kinds of application software supplied at appropriate times and image data picked up by the CCD camera 8 to the liquid crystal display 10 of the display unit 3,
A plurality of window screens can be displayed.

IEEE 1349 interface 57
Is directly connected to the IEEE 1394 terminal 28, and can be connected to another computer device or an external device such as a digital video camera via the IEEE 1394 terminal 28.

When an optional function is added, the PC card interface 58
7 is connected to a PC card (not shown) mounted on the PC 7 and, for example, a CD-ROM (Compact Dimm
sc-Read Only Memory) drive and DVD (Digital Ve
(rsatile Disc) drive so that it can be connected to an external device.

The PCI bus 55 is an ISA (Industrial S)
and a PCI-ISA bridge 66 and a PCI-ISA bridge 66.
The HDD 67 and the USB terminal 21 are connected to the bridge 66.

Here, the PCI-ISA bridge 66
DE (Integrated Drive Electronics) interface, configuration register, RTC (Real-T
imeClock) circuit, a USB interface, and the like, and controls the HDD 67 via the IDE interface based on a system clock supplied from the clock generator 60.

The hard disk of the HDD 67 has Windo
OS (Operating System) such as ws98 (trademark), e-mail program, auto pilot program, jog dial server program, jog dial driver,
Capture software, digital map software, and various other application software are stored, and are transferred to and loaded on the RAM 53 as needed in the course of startup processing.

The PCI-ISA bridge 66 is a U.S.A.
External devices such as a floppy (registered trademark) disk drive, a printer, and a USB mouse (not shown) connected via the B terminal 21 are controlled via a USB interface, and a modem 69 and a sound controller 70 connected to the ISA bus 65. Control.

The modem 69 is connected to an Internet service provider (hereinafter, referred to as a provider) from a modem terminal 25 via a public telephone line (not shown), and is connected to the Internet via the provider through a dial-up IP connection. ing.

The sound controller 70 generates audio data by digitally converting the audio signal collected by the microphone 13, outputs the audio data to the CPU 50, and converts the audio data supplied from the CPU 50 into an analog signal. , And outputs it to the outside via the built-in speaker 6.

The ISA bus 65 has an I / O (In / Ou).
t) The controller 73 is connected, receives power from an external power supply from the external power supply connector 22 via the power supply charging control circuit 85, and supplies power to each circuit when the power switch 7 is turned on. Here, the I / O controller 73 operates based on the system clock supplied from the clock generator 60 also.

The power supply charge control circuit 85 is provided with an I / O
The controller 73 controls charging of the battery pack 30 connected to the battery connector 29 (FIG. 3).

The I / O controller 73 includes a microcontroller, an I / O interface, a CPU, a ROM,
The flash memory 7 comprises a RAM or the like.
9 (Basic Input / Output Sys.)
tem) to control the input and output of data between the OS and application software and various peripheral devices such as the liquid crystal display 10 and the HDD 67.

The I / O controller 73 is connected to the infrared port 18 and can execute infrared communication with, for example, another computer.

Further, the I / O controller 73 is connected to an inversion switch 77, and when the imaging unit 11 is rotated by 180 degrees toward the back side of the liquid crystal display 10, the inversion switch 77 is turned on. -ISA
The CPU 50 is notified via the bridge 66 and the host-PCI bridge 54.

In addition, the I / O controller 73
It is connected to the full-press / half-press switch 78 and the main body 2
When the shutter button 9 provided on the upper surface of the camera is half-pressed, the full-press / half-press switch 78 is turned on to the half-pressed state, the CPU 50 is notified of this, and the shutter button 9 is fully pressed. The full-press / half-press switch 78 is turned on to the full-press state, and the CPU 5
Notify 0.

That is, the CPU 50 transfers the capture software from the hard disk of the HDD 67 to the RAM 53.
When the shutter button 9 is half-pressed by the user in a state where the camera is raised up, the still image mode is entered and the CC
The D camera 8 is controlled to freeze the still image. When the D camera 8 is fully pressed, the frozen still image data is captured and sent to the video controller 56.

On the other hand, when the shutter button 9 is fully depressed by the user without starting the capture software, the CPU 50 enters the moving image mode, captures a moving image for up to about 60 seconds, and outputs the moving image. The data is sent to the controller 56.

By the way, the RO of the I / O controller 73
M stores a wakeup program, a key input monitoring program, an LED control program, a jog dial state monitoring program, and various other control programs.

Here, the jog dial state monitoring program is a program used in conjunction with a jog dial server program stored on the hard disk of the HDD 67, and monitors whether the jog dial 24 is rotated or pressed. It is.

The wake-up program is a PCI-I
When the current time supplied from the RTC circuit in the SA bridge 66 coincides with a preset start time, the CPU 50 is a program controlled to execute a predetermined process. Is a program for monitoring inputs from various key switches. The LED control program includes a power lamp PL,
This is a program for controlling lighting of various lamps such as a battery lamp BL and a message lamp ML (FIG. 1).

In the RAM of the I / O controller 73, an I / O register for a jog dial state monitoring program, a set time register for a wakeup program, a key input monitoring register for a key input monitoring program, and an LED control program An LED control register and other registers for various programs are provided.

The set time register stores the time information of the start time arbitrarily set by the user in advance for use in the wake-up program. Accordingly, the I / O controller 73 determines whether or not the current time supplied from the RTC circuit matches the arbitrarily set start time based on the wake-up program. Notify.

Thus, the CPU 50 starts up predetermined application software set in advance at the start time, and executes predetermined processing according to the application software.

The key input monitoring register stores operation key flags corresponding to input operations of the operation key 4, the stick 5, the left click button 5A, the right click button 5B, the center button 5C and the like.

Therefore, the I / O controller 73 operates based on the key input monitoring program to determine whether or not the pointing operation with the stick 5 or the clicking operation of the left click button 5A, right click button 5B and center button 5C has been performed. The determination is made based on the state of the key flag, and when a pointing operation or a click operation is performed, the fact is notified to the CPU 50.

Here, the pointing operation is an operation of moving the mouse cursor to a desired position on the screen by pressing the stick 5 up, down, left and right with a finger, and the clicking operation is the left click button 5A or the right click button. This is an operation to quickly press and release the click button 5B with a finger.

Thus, the CPU 50 executes a predetermined process in accordance with the movement of the mouse cursor and the click operation by the pointing operation.

The LED control register includes a power lamp P
A lighting flag indicating the lighting state of various lamps such as L, battery lamp BL, and message lamp ML is stored.

Therefore, the I / O controller 73 causes the CPU 50 to operate the HD
The e-mail program is started from the hard disk at D67, and when an e-mail is received according to the e-mail program, a lighting flag is stored, and the message lamp ML is turned on by controlling the LED 81 based on the lighting flag.

The I / O register for the jog dial state monitoring program stores a rotation operation flag and a pressing operation flag corresponding to a rotation operation and a pressing operation on the jog dial 24.

Therefore, when a user's desired menu item is selected from a plurality of menu items by rotating and pressing the jog dial 24 connected via the rotation detecting unit 88, the I / O controller 73 outputs the I / O controller 73. The rotation operation flag and the pressing operation flag stored in the O register are set, and the CPU 50 is notified to that effect.

Thus, according to the jog dial server program read out from the HDD 67 and started on the RAM 53, the CPU 50 starts up application software corresponding to the menu item determined by rotating and pressing the jog dial 24, and performs a predetermined process. Execute.

Here, the I / O controller 73 operates constantly under the control of the power supply / charge control circuit 85 even when the power switch 7 is off and the OS is not running. The user can start application software or a script file desired by the user by pressing the jog dial 24 in the power saving state or when the power is off.

[0058] Note that the I / O controller 73 is also connected to the I ² C bus 82, supplies the various setting parameters for the CCD camera 8 set by operating keys 4 or the jog dial 24 through the I ² C bus 82 Thus, the brightness and contrast of the CCD camera 8 are adjusted.

(1-3) Gesture Recognition Processing In addition to the above configuration, the notebook personal computer 1 uses the HDD 67 with application software called a cyber gesture program for recognizing a user's hand movement (gesture) captured by the CCD camera 8. From the hard disk, recognizes the movement of the user's hand captured by the CCD camera 8 based on the cyber gesture program, and executes a predetermined process according to the recognition result on the active window screen based on the application software. Has been made.

That is, the notebook personal computer 1 starts up an image editing program capable of processing a captured still image, for example, and selects a plurality of images stored in the hard disk of the HDD 67 in order to select a still image to be processed. When the still images are sequentially displayed on the liquid crystal display 10, an image feed operation is performed such that the still images displayed on the liquid crystal display 10 are moved forward or backward one by one in response to the turning operation of the jog dial 24 by the user. In the present invention, by making the CPU 50 recognize the movement of the user's hand captured by the CCD camera 8, the above-described image feed operation can be executed without touching the jog dial 24 under the control of the CPU 50. ing.

Incidentally, in the notebook computer 1, one still image displayed on the liquid crystal display 10 when the jog dial 24 is rotated more than a predetermined angle to the depth side as viewed from the user is displayed.
When the jog dial 24 is rotated by a predetermined angle or more toward the user as viewed from the user, only one still image displayed on the liquid crystal display 10 is fed back.

In practice, the CPU 50 of the notebook computer 1
Entering from the start step of the routine RT1 in FIG. 5, the process proceeds to the next step SP1, and the HDD 67 is operated according to a user operation.
After launching a cyber gesture program from the hard disk of the above, a gesture recognition screen 100 as shown in FIG. 6 is generated according to the cyber gesture program, and this is superimposed and displayed on a still image of an active window screen according to the image editing program. Move to the next step SP2.

Here, as shown in FIG. 7, the gesture recognition screen 100 has a screen size of 164 × 136 pixels, and “CYBERGESTURE” (Cyber Gesture Program) is displayed at the upper end of the gesture recognition screen 100 to indicate that it is a cyber gesture program. Sony Corporation trademark title part 101, option button 102 for selecting option of function, help button 103, minimize button 1
04 and a close button 105 are provided.

The gesture recognition screen 100 is formed with an extremely small screen size as compared with the screen size (1024 × 480 pixels) of the liquid crystal display 10, whereby the active state displayed on the background of the gesture recognition screen 100 is displayed. The concealment area of a window screen from a still image is minimized.

Note that the CPU 50 of the notebook personal computer 1
02, when the mouse cursor is placed on any one of the help button 103, the minimize button 104, and the close button 105, the button is displayed in a raised state, and after the clicked selection, the button is recessed. Is displayed, so that selection and determination operations on the button portion can be easily performed visually.

The CPU 50 of the notebook personal computer 1 displays the gesture recognition display area 106 on the gesture recognition screen 100 in a gray scale of 256 gradations, and has five squares at the approximate center of the gesture recognition display area 106. The target portions 107 formed by the targets 107A to 107E are arranged in a row and displayed.

Thus, the CPU 50 allows the notebook computer 1 to recognize the left-right movement when the user's hand is moved by the target unit 107 displayed in the gesture recognition display area 106 of the gesture recognition screen 100. The user can be easily imagined that the user is informed.

Each of the targets 107A to 107E is
As shown in FIG. 8, the frame portions 107AF to 10AF each having a size of 8 pixels × 8 pixels and each having a width of 1 pixel.
7EF and the frame portions 107AF-
107EF is displayed in red,
This makes it easier to visually recognize the targets 107A to 107E with respect to the gray scale display of the background.

In the gesture recognition display area 106 of the gesture recognition screen 100, a black line display (not shown) is provided for every two horizontal lines of the scanning lines, thereby providing a screen for displaying a normal image. In contrast, the gesture recognition screen 100 is made to be easily recognized by the user.

In step SP 2, the CPU 50 images the user present in front of the display unit 3 with the CCD camera 8 of the imaging unit 11, and obtains the resulting input image on the gesture recognition display area 10 of the gesture recognition screen 100.
6 and proceed to the next subroutine SRT2.

As shown in FIG. 9, the subroutine SRT2
In step SP21, the CPU 50 determines the gesture recognition display area 106 of the gesture recognition screen 100.
Is divided into a plurality of types of color regions based on the color components, and the process proceeds to the next step SP22.

Incidentally, the color area is represented on a predetermined YUV chromaticity space as shown in FIG. 10, and for example, a predetermined area of + Y, -U, and -V quadrants indicated by oblique lines in the YUV chromaticity space. Is regarded as a color region R equivalent to the palm color of the user (hereinafter, referred to as a skin color region R).

At step SP22, the CPU 50
The predetermined skin color table corresponding to the skin color region R on the YUV (luminance / color difference) chromaticity space is compared with each color region of the input image, and the process proceeds to the next step SP23.

In this case, each color region of the input image is roughly divided into a skin color region R such as a user's face region and palm region, and a non-skin color region of a clothing portion.

At step SP23, the CPU 50
As a result of comparing the skin color table with each color region of the input image, it is determined whether or not a skin color region R recognized as a skin color exists in the input image.

If a negative result is obtained here, it means that the skin color region R corresponding to the skin color table does not exist in the input image.
Moves to the next step SP29.

At step SP29, the CPU 50
Since the skin color region R does not exist in the input image and the movement of the user's hand cannot be recognized, the process proceeds to the input image of the next frame, and returns to step SP21 described above.

On the other hand, if a positive result is obtained in step SP23, this means that the skin color region R corresponding to the skin color table exists in the input image. At this time, the CPU 50 Move to step SP24.

In step SP24, the CPU 50
The movement of the skin color region R in the input image of the current frame is detected based on a change in the coordinate values between the current frame and the previous frame, and the flow advances to next step SP25.

At step SP25, the CPU 50
It is determined whether a moving skin color region R exists in the input image. If a negative result is obtained here, this indicates that the moving skin color region R does not exist in the input image. At this time, the CPU 50 proceeds to the next step SP29, and inputs the next frame. The process proceeds to the image and returns to step SP21 described above.

On the other hand, if an affirmative result is obtained in step SP25, this means that a moving skin color region R exists in the input image.
The PU 50 moves to the next step SP26.

At step SP26, the CPU 50
After detecting the largest skin color region R among the moving skin color regions R and temporarily determining this as a palm region, the process proceeds to the next step SP27.

At step SP27, the CPU 50
The coordinate values of the entire palm area determined in step SP26 are obtained, and the routine goes to the next step SP28.

At step SP28, the CPU 50
After calculating the center of gravity of the palm area based on the coordinate values of the entire palm area acquired in step SP27, the coordinates of the upper end position of the palm area corresponding to the vertical direction above the center of gravity are detected, and this is set as the fingertip of the hand. The subroutine SRT2 is obtained by acquiring the corresponding data above the center of gravity.
Ends the acquisition processing procedure regarding hand position information in
The process moves to step SP3 of the routine RT1 (FIG. 5).

At step SP 3, the CPU 50 determines whether or not the user's hand is in the gesture recognition display area 106 of the gesture recognition screen 100 based on the data on the center of gravity acquired in the subroutine SRT 2.

If a negative result is obtained here, this means that the data above the center of gravity has not been obtained in the subroutine SRT2, that is, the user's hand is
0 indicates that it does not exist in the gesture recognition display area 106. At this time, the CPU 50 proceeds to the next step SP4.

In step SP 4, since the user's hand is not displayed in the gesture recognition display area 106 of the gesture recognition screen 100, the CPU 50 performs an animation display indicating that it is currently in a search state for recognizing the user's hand. The process returns to step SP2.

In this case, as shown in FIG.
Indicates that the user's skin color portion is hardly displayed in the gesture recognition display area 106 of the gesture recognition screen 100, and the user's hand cannot be recognized at this time.
The target state 10 indicates that the skin color region R is being searched.
7 allows the user to easily recognize the animation.

That is, the CPU 50 controls each target 10
Frame portions 107AF to 107EF in 7A to 107E
Are displayed alternately in red along the left and right directions indicated by arrows A and B in order to produce a gradation effect. It is designed to make it easy for the user to imagine something.

On the other hand, if an affirmative result is obtained in step SP3, this means that the data above the center of gravity has been obtained in the subroutine SRT2, that is, the user's hand is in the gesture recognition display area 10 of the gesture recognition screen 100.
6 that the CPU 5
"0" moves to the next step SP5.

In step SP5, the CPU 50 displays a pointer 108 having a predetermined shape at a position corresponding to the acquired data on the center of gravity as shown in FIG. 12, and surrounds the entire palm area of the user including the pointer 108. The overlapped palm area recognition frame 109 is displayed on the input image of the gesture recognition display area 106,
Move to the next subroutine SRT3.

Here, the CPU 50 sets the palm area recognition frame 1
09 is displayed in white with one pixel width, and the target unit 107 is displayed.
A pointer frame 108F having a width of one pixel of the pointer 108 formed in the same shape and size as each of 107A to 107E is displayed in white, and the inside thereof is displayed in red.

Thus, the CPU 50 sets each target 1
Frame portions 107AF to 107E in 07A to 107E
F in red and the pointer frame 1 in the pointer 108
08F, each target 107A-
107E and the pointer 108 are clearly distinguished for the user.

The CPU 50 displays the palm area recognition frame 109 and the pointer 108 while moving them in conjunction with the movement of the user's hand.

Subsequently, as shown in FIG. 13, in step SP31 of the subroutine SRT3, the CPU 50
The fingertip moving distance is obtained based on the difference between the coordinate values of the data above the center of gravity between the current frame stored in the ring buffer shape using AM53 and the previous frame adjacent to the current frame, and the next step SP32 Move on to

At step SP32, the CPU 50
It is determined whether or not the fingertip movement distance between adjacent frames calculated in step SP31 is equal to or less than a predetermined upper threshold. If a negative result is obtained here, this means that the distance from the position indicating the fingertip of the previous frame to the position indicating the fingertip of the current frame is extremely far away, and as data for recognizing hand movement. This indicates that the fingertip movement distance is inappropriate, and at this time, the CPU 50 proceeds to the next step S
Move to P33.

[0097] In step SP33, the CPU 50
Since it is inappropriate to use the fingertip movement distance between adjacent frames as data, the calculation of the fingertip movement distance performed after step SP34 is stopped, and the routine RT1 (FIG. 5)
Returning to step SP2, the above processing is repeated.

On the other hand, if a positive result is obtained in step SP32, this means that the distance from the position indicating the fingertip of the previous frame to the position indicating the fingertip of the current frame is not extremely far, and the hand movement This indicates that the fingertip movement distance is appropriate as data for recognizing, and the CPU 50 proceeds to the next step SP34.

In step SP34, the CPU 50
As shown in FIG. 14, upper center-of-gravity data indicating the fingertip of the current frame sequentially stored in a ring buffer shape, and upper center-of-gravity data indicating the fingertip of an arbitrary past frame selected from several past frames within a predetermined range. The difference between the coordinate values having the maximum length is calculated as the maximum fingertip movement distance, and it is determined whether the maximum fingertip movement distance is greater than a predetermined lower threshold.

If a negative result is obtained here, this means that the maximum fingertip movement distance based on the state transition of the input image over a plurality of frames is smaller than a predetermined lower threshold, that is, it is not worth recognizing hand movement. At this time, the CPU 50 excludes the maximum fingertip movement distance from the recognition processing, returns to step SP31, and repeats the above processing.

On the other hand, if an affirmative result is obtained in step SP34, this means that the maximum fingertip movement distance is larger than the predetermined lower limit threshold, and it is recognized that the fingertip of the hand has surely moved left and right. At this time, the CPU 50 moves to the next step SP35.

In step SP35, the CPU 50
The moving direction (rightward or leftward) of the fingertip moving distance based on the moving vector of the data above the center of gravity indicating the fingertip of the current frame and the data above the center of gravity indicating the fingertip of the past frame used when calculating the maximum fingertip moving distance. ), And returns to step SP6 of the routine RT1 (FIG. 5).

In step SP6, the CPU 50 detects the maximum fingertip movement distance and the movement direction of the maximum fingertip movement distance, so that the CPU 50 calculates the coordinate values of the pixel data of the current frame and the pixel data of the previous frame in the entire detected palm area. Based on the change per unit time, it is determined whether the detected movement speed of the entire palm area exceeds a predetermined speed.

If a negative result is obtained here, this means that the detected movement speed of the entire palm region does not exceed a predetermined speed, that is, since the movement is relatively slow, the actual palm region is not the palm region but the face region. It is determined that there is a possibility, and the process returns to step SP2 again to repeat the above processing.

On the other hand, if a positive result is obtained in step SP6, this means that the detected motion speed of the entire palm area exceeds a predetermined speed, that is, the palm area is moving relatively fast, and thus the palm area may be a palm area. It is determined that the property is even higher, and the routine goes to the next step SP7.

In this case, when there are two or more palm area candidates in the input image of a single frame, the CPU 50 can more accurately discriminate between the palm area and the face area.

At step SP7, the CPU 50 executes the cyber gesture program 18 as shown in FIG.
The recognition result of the gesture motion of the palm area recognized based on the O.D.
The trajectory representing the hand movement (gesture) moved by the user and the recognition process indicating how the notebook computer 1 recognized the gesture are supplied to the jog dial server program 182 via the gramming interface 181. Visual feedback is displayed on the gesture recognition screen 100, and the next step SP8
Move on to

Here, the API is a program interface open to the application software by the OS, and the application software basically performs all processing via the API. Incidentally, the API of the general OS at present takes the form of a function, and the function of the API is called from application software by designating appropriate arguments (parameters).

Incidentally, the CPU 50 operates the jog dial 24.
The operation result and the recognition result based on the cyber gesture program 180 are fetched in the same input format, and supplied to the jog dial server program 182 via the shared jog dial API 181. This simplifies software processing. It has been made possible.

In practice, the CPU 50 generates a visual feedback screen 191 as shown in FIG.
Via the visual feedback screen 191, the pointer 10 is placed on targets 107A to 107E that are arranged obliquely in advance in the trajectory display frame 120 in accordance with the trajectory indicating the hand movement (gesture) actually moved by the user.
By superimposing 8 and displaying it while moving it in the direction of arrow C, the user can visually confirm the actual hand movement recognition process.

Subsequently, the CPU 50 generates a visual feedback screen 192 as shown in FIG.
This is replaced with the visual feedback screen 191 and displayed.

The visual feedback screen 192
Is the target 107A by deforming the trajectory display frame 120 in the visual feedback screen 191.
Direction display frame 1 with heights of ~ 107E aligned in a horizontal row
21 is formed and linearized, and the direction display frame 12 is formed.
1, the target 107E is at the right end, and the pointer 1 is at the left end.
08 is displayed, and the direction display frame 1 is displayed.
It is shown simply that the user's hand has been moved in the direction of arrow D (from right to left) via 21.

Finally, the CPU 50 generates a visual feedback screen 193 as shown in FIG.
This is replaced with the visual feedback screen 192 and displayed.

The visual feedback screen 193
Deletes the direction display frame 121 of the visual feedback screen 192 and repeatedly displays the pointer 108 on each of the targets 107A to 107E arranged in a horizontal line while moving the pointer 108 in the direction of arrow D. The user can easily recognize that the notebook computer 1 has recognized that the user's hand has been moved from the right side to the left side (the direction of arrow D).

The CPU 50 controls each target 107A.
When the pointer 108 is superimposed on to 107E and displayed while being moved in the direction of arrow D, the pointer 108 is moved at the same moving speed as when the movement of the user's hand is recognized. Thus, the user can be informed of the moving speed of the hand recognizable by the notebook computer 1.

In step SP8, after recognizing the movement of the user's hand, the CPU 50 issues a predetermined command corresponding to the movement of the hand to the jog dial server program 18.
2 (FIG. 15) to application software 183
And performs a predetermined process according to the recognition result. At this time, the gesture recognition process is performed on the input image for several frames immediately after the recognition of the hand movement. Instead, the process returns to step SP2 and repeats the above processing.

Thus, the CPU 50 executes the processing corresponding to the movement of the user's hand on the active window screen without fail, and then executes the processing corresponding to the movement of the next user's hand. Can be.

Thus, the CPU 50 of the notebook personal computer 1
After recognizing the movement of the user's hand in accordance with the cyber gesture program 180, the jog dial server program 18
2 to the application software 183 to execute a predetermined image feed operation according to a command on an active window screen based on the application software 183.

In practice, the CPU 50 of the notebook personal computer 1
When the gesture of the hand is recognized as moving from left to right (the direction opposite to the direction of arrow D), the still image of the active window screen displayed on the background of the gesture recognition screen 100 is advanced by one image. When the gesture of the hand is recognized as a movement from the right side to the left side (in the direction of arrow D), the still image of the active window screen displayed on the background of the gesture recognition screen 100 is returned by one frame. ing.

Thus, the user can enter the jog dial 24
Without directly operating the CCD camera 8 of the imaging unit 11.
The still image of the active window screen displayed on the background of the gesture recognition screen 100 can be image-forwarded in a desired direction only by moving the hand to the left or right while holding the hand.

(1-4) Operation and Effect of First Embodiment In the above configuration, the CPU 50 of the notebook personal computer 1
Activates the cyber gesture program 180 with the image editing program started and the active window screen displayed on the liquid crystal display 10, thereby displaying the gesture recognition screen 100 over the still image of the active window screen.

Then, the CPU 50 of the notebook computer 1
The user located in front of the display unit 3 is connected to the CCD of the imaging unit 11
The camera 8 picks up an image and displays the resulting input image on the gesture recognition display area 1 of the gesture recognition screen 100.
In addition to the display of the command at 06, the movement of the user's hand is recognized, and a command is generated in accordance with the recognized movement of the hand.

At this time, the CPU 50 of the notebook personal computer 1
Is the hand movement (gesture) actually moved by the user
16A, the targets 107A to 107A are tilted in advance in the locus display frame 120 as shown in FIG.
07E, the pointer 108 is superimposed on the targets 107A to 107E, and alternately displayed in order while moving in the direction of arrow C, so that the trajectory of the recognition process for the movement of the user's hand is visually recognized by the user. Can be confirmed.

Thereafter, the CPU 50 forms the direction display frame 121 in which the heights of the targets 107A to 107E are aligned in a horizontal line by deforming the trajectory display frame 120 as shown in FIG. In addition, the target 107E is displayed at the right end in the direction display frame 121 and the pointer 108 is displayed at the left end in the direction display frame 121, thereby simply indicating that the user's hand has been moved in the direction of arrow D (left direction).

Finally, the CPU 50 erases the direction display frame 121 as shown in FIG. 16 (C), overlaps the pointer 108 on each of the targets 107A to 107E, and uses the moving speed of the user's hand at the time of recognition. By displaying the pointer 108 while moving it in the direction of arrow D, it is possible to reliably notify the user that the user's hand has been recognized to have been moved from right to left (in the direction of arrow D).

According to the above configuration, the notebook computer 1 displays the trajectory and the recognition result of the recognition process for the movement of the user's hand picked up by the CCD camera 8 in each of the targets 107A to 107A.
By displaying the animation in an easy-to-understand manner using the pointer 7E and the pointer 108, it is possible to feed back and learn how the movement of the hand of the user was recognized in what kind of recognition process.

Thus, the user causes the notebook computer 1 to recognize the movement of the hand in consideration of the moving direction and the moving speed of the hand, so that, for example, the input operation of the command for performing the image feed operation can be performed in a short time. Can be performed.

(2) Second Embodiment (2-1) Overall Configuration of Network System In FIG. 17, reference numeral 200 denotes a network system to which a mobile phone MS3 to which the present invention is applied is connected as a whole. Base station CS, which is a fixed radio station, in a cell obtained by dividing the service area of
1 to CS4 are provided.

The base stations CS1 to CS4 include mobile information terminals MS1 and MS2 as mobile radio stations and digital mobile phones MS3 and MS4 with cameras, for example, WC.
Wireless connection is made by a code division multiple access method called DMA (Wideband-Code Division Multiple Access), and a large capacity at a maximum data transfer rate of 2 [Mbps] using a frequency band of 2 [GHz]. Data is transmitted at high speed.

As described above, the portable information terminals MS1 and MS2
And camera-equipped digital mobile phones MS3 and MS4
The W-CDMA system enables high-speed data communication of large-volume data, so it can perform not only voice calls but also various data communications such as sending and receiving e-mails, browsing simple homepages, and sending and receiving images. It is made to be able to do.

The base stations CS1 to CS4 are connected to a public line network INW via a wired line. The public line network INW includes the Internet ITN, many subscriber wired terminals (not shown), computer networks, and corporate networks. The internal network is connected.

An access server AS of an Internet service provider is also connected to the public network INW, and a content server TS owned by the Internet service provider is connected to the access server AS.

[0133] The content server TS converts content such as a simple homepage into, for example, a compact HTML (Hyper Text Markup Language) in response to a request from a subscriber wired terminal or mobile information terminal MS1, MS2 and a digital mobile phone with camera MS3, MS4. ) Format file.

Incidentally, a number of WWW servers WS1 to WSn are connected to the Internet ITN,
Subscriber wired terminal or portable information terminal M according to P protocol
S1, MS2 and digital mobile phone MS with camera
3. The MS 4 can access the WWW servers WS1 to WSn.

The portable information terminals MS1 and MS2 and the digital cellular phones with cameras MS3 and MS4 communicate with base stations CS1 to CS4 (not shown) using a simple transport protocol of 2 [Mbps].
WWW server W from S4 via Internet ITN
S1 to WSn are communicated by the TCP / IP protocol.

The management control unit MCU is connected to the public network I
Subscriber wired terminal or portable information terminal MS1, M via NW
S2 and digital mobile phone with camera MS3, MS4
And the subscriber's wired terminal, the portable information terminals MS1, MS2, and the camera-equipped digital cellular phone MS.
3. Authentication processing, charging processing, and the like for the MS 4 are performed.

(2-2) External Configuration of Digital Mobile Phone with Camera Next, the digital mobile phone with camera M to which the present invention is applied
The external configuration of S3 will be described. As shown in FIG. 18, the digital mobile phone with camera MS3 is divided into a display unit 212 and a main body 213 with a central hinge 211 as a boundary, and is foldable via the hinge 211.

On the display section 212, a transmitting / receiving antenna 214 is attached to the upper left portion so as to be able to be pulled out and stored, and the base station CS3 is connected via the antenna 214.
It is designed to transmit and receive radio waves between and.

The display section 212 is provided with a camera section 215 which is rotatable at an angle range of about 180 degrees at the center of the upper end, and the CCD camera 216 of the camera section 215 is provided.
Thus, a desired imaging target can be imaged.

Here, when the camera unit 215 is rotated by approximately 180 degrees and positioned by the user, the display unit 212 is provided with a speaker 217 provided at the center of the rear side of the camera unit 215 as shown in FIG. Side, thereby switching to a normal voice call state.

Further, the display unit 212 is provided with a liquid crystal display 218 in front of the display unit 212, for receiving radio wave reception status, remaining battery power, a destination name and telephone number registered as a telephone directory, a transmission history, and the like. The content of the e-mail, the simple homepage, and the image captured by the CCD camera 216 of the camera unit 215 can be displayed.

On the other hand, the main body 213 has “0” on its surface.
There are provided operation keys 219 such as a number key, a call key, a redial key, an end call and power key, a clear key, an e-mail key, and the like of "9", and various instructions are inputted using the operation keys 219. Have been made to gain.

The main body 213 is provided with a memo button 220 and a microphone 221 below the operation keys 219, so that the memo button 220 can record the voice of the other party during a call, and the microphone 221 can be used to record a call. The sound of the user is collected.

Further, the main body 213 is provided with a rotatable jog dial 222 above the operation keys 219.
3 is provided so as to protrude slightly from the surface,
Various operations such as a scrolling operation of a telephone directory list or an e-mail displayed on the liquid crystal display 218, a page turning operation of a simple homepage, an image sending operation, and the like are performed in response to the turning operation on the jog dial 222. ing.

For example, the main body 213 is turned on by the user by turning the jog dial 222 by the user.
A desired telephone number is selected from a plurality of telephone numbers in the telephone directory list displayed at 18 and the jog dial 22 is selected.
When the user presses the button 2 toward the inside of the main body 213, the selected telephone number is determined and the calling process is automatically performed on the telephone number.

The main body 213 has a battery pack (not shown) inserted on the back side, and when the call is ended and the power key is turned on, power is supplied from the battery pack to each circuit portion, and the main body 213 can operate. Start up in a proper state.

The main body 213 has the main body 213.
A memory stick slot 224 for inserting a detachable memory stick (trademark of Sony Corporation) 223 is provided on the upper left side of the device. When the memo button 220 is pressed, a call can be made to the memory stick 223 during a call. Record the voice of the other party, send e-mail,
A simple homepage is provided so that images captured by the CCD camera 216 can be recorded.

Here, the memory stick 223 is a type of flash memory card developed by Sony Corporation, the present applicant. This memory stick 2
Reference numeral 23 denotes an EEPROM (Electrically Era) which is a nonvolatile memory which can be electrically rewritten and erased in a small and thin plastic case having a size of 21.5 × 50 × 2.8 [mm].
sable and Programmable Read Only Memory), which stores flash memory elements.
Various data such as images, sounds, and music can be written and read via pin terminals.

The memory stick 223 can be used in response to a change in the specification of the built-in flash memory due to an increase in capacity or the like.
Adopts a unique serial protocol that can ensure compatibility with the device used, with a maximum writing speed of 1.5 [MB / S]
It achieves high-speed performance with a maximum read speed of 2.45 [MB / S] and secures high reliability by providing an erasure prevention switch.

Therefore, the camera-equipped digital mobile phone MS
3 is configured to be able to insert such a memory stick 223, so that data can be shared with another electronic device via the memory stick 223. .

(2-3) Circuit Configuration of Digital Mobile Phone with Camera As shown in FIG.
In step S3, the power supply circuit unit 251, the operation input control unit 252, the image encoder 253, the camera interface unit 254, and the LCD control unit 250 control the main control unit 250 that controls the display unit 212 and the main body 213 in an integrated manner. (Li
quid Crystal Display) control unit 255, image decoder 2
56, a demultiplexing unit 257, a recording / reproducing unit 262, a modulation / demodulation circuit unit 258, and an audio codec 259.
0 and connected to each other via the image encoder 2
53, an image decoder 256, a demultiplexing unit 257, a modulation / demodulation circuit unit 258, and an audio codec 259.
1 are connected to each other.

When the call end and power key are turned on by the user's operation, the power supply circuit section 251 supplies power to each section from the battery pack to enable the digital cellular phone with camera MS3 to operate. To start.

The digital mobile phone with camera MS3 is
Under the control of the main control unit 250 including a CPU, a ROM, a RAM, and the like, the
1 is converted into digital audio data by the audio codec 259, and this is converted into a digital demodulation circuit unit 2.
The spread spectrum processing is performed at 58, and the digital signal is subjected to digital / analog conversion processing and frequency conversion processing at the transmission / reception circuit section 262, and then transmitted via the antenna 214.

A digital mobile phone with camera MS3
After amplifying the received signal received by the antenna 214 in the voice communication mode, performing a frequency conversion process and an analog-to-digital conversion process, performing a spectrum despreading process in the modulation / demodulation circuit unit 258, and converting the received signal into an analog voice signal by the voice codec 259. Is output via the speaker 217.

Further, digital mobile phone MS with camera
When transmitting an e-mail in the data communication mode, the e-mail 3 sends text data of the e-mail input by operating the operation keys 219 and the jog dial 222 to the main control unit 250 via the operation input control unit 252.

The main control section 250 subjects the text data to spread spectrum processing in the modulation / demodulation circuit section 258, performs digital / analog conversion processing and frequency conversion processing in the transmission / reception circuit section 262, and then passes the base station CS 3 via the antenna 214.
(FIG. 17).

On the other hand, when receiving an e-mail in the data communication mode, the camera-equipped digital mobile phone MS3 performs a spectrum despreading process on the received signal received from the base station CS3 via the antenna 214 in the modulation / demodulation circuit unit 258. After the original text data is restored, the data is displayed as an electronic mail on the liquid crystal display 218 via the LCD control unit 255.

Thereafter, a digital mobile phone with a camera MS
3 can also record an e-mail received in response to a user operation on the memory stick 223 via the recording / reproducing unit 262.

On the other hand, camera-equipped digital mobile phone MS3
When transmitting image data in the data communication mode,
The image data captured by the CCD camera 216 is transferred to the image encoder 253 via the camera interface unit 254.
To supply.

By the way, digital mobile phone MS with camera
3 does not transmit image data, it is also possible to directly display image data captured by the CCD camera 216 on the liquid crystal display 218 via the camera interface unit 254 and the LCD control unit 255.

The image encoder 253 is a CCD camera 2
The image data supplied from the MPU 16 is, for example, MPEG (Movi)
g Picture Experts Group) 2, converts the encoded image data into encoded image data by compression encoding according to a predetermined encoding method such as MPEG4, and sends it to the demultiplexing unit 257.

At the same time, the camera-equipped digital mobile phone MS3 simultaneously converts the sound collected by the microphone 221 during imaging by the CCD camera 216 as digital sound data via the sound codec 259 into the demultiplexing unit 257.
To send to.

The demultiplexing unit 257 includes the image encoder 25
3 and the coded image data supplied from 3 and the audio codec 2
The multiplexed data is multiplexed with the audio data supplied from 59 in a predetermined manner, and the resulting multiplexed data is
Then, the transmission / reception circuit unit 262 performs a digital-to-analog conversion process and a frequency conversion process, and then transmits the signal via the antenna 214.

On the other hand, in the case of receiving image data such as a simple homepage in the data communication mode, the camera-equipped digital mobile phone MS3 modulates the received signal received from the base station CS3 via the antenna 214 by the modem circuit 258. The spectrum despreading process is performed, and the resulting multiplexed data is sent to the demultiplexing unit 257.

The demultiplexing section 257 separates the multiplexed data into coded image data and audio data, supplies the coded image data to the image decoder 256 via the synchronous bus 261 and outputs the coded data. Is supplied to the audio codec 259.

The image decoder 256 generates reproduced image data by decoding the encoded image data by a decoding method corresponding to a predetermined encoding method such as MPEG2 or MPEG4, and outputs the reproduced image data via the LCD control unit 255. The image is displayed on the liquid crystal display 218 as an image linked to, for example, a simple homepage.

At this time, the audio codec 259 simultaneously
After converting the audio data into an analog audio signal, the audio data is output via the speaker 217 as, for example, audio linked to a simple homepage.

In this case, similarly to the case of the electronic mail, the digital mobile phone with camera MS3 can record the received image data of the simple homepage on the memory stick 223 via the recording / reproducing unit 262 by the operation of the user. is there.

In addition to the above configuration, the camera-equipped digital mobile phone MS3 stores the same cyber gesture program 18 in the ROM of the main controller 250 as in the first embodiment.
0 (FIG. 15) and the jog dial server program 18
2 is stored, and the gesture recognition screen 100 (FIG. 6) is superimposed on the active window screen based on the cyber gesture program 180 in a state where the active window screen based on the predetermined application software 183 is displayed on the liquid crystal display 218. And the CCD camera 216 is displayed in the gesture recognition display area 106 of the gesture recognition screen 100.
It is possible to display the image of the user captured by the user.

Next, the digital cellular phone MS3 with a camera
As in the first embodiment, as shown in FIGS. 5 to 16, the skin color area is selected from the user's image displayed in the gesture recognition display area 106 of the gesture recognition screen 100 under the control of the main control unit 250. After detecting R, and recognizing the moving skin color region R as a palm region,
A predetermined command corresponding to the gesture operation of the palm area is supplied to the application software 183 via the jog dial server program 182.

As a result, the camera-equipped digital mobile phone MS3 is connected to the notebook personal computer 1 of the first embodiment.
Similarly, the gesture recognition screen 1 according to the application software 183 under the control of the main control unit 250.
The still image of the active window screen displayed on the background of the image No. 00 can be fed in a desired direction in response to a command.

(2-4) Operation and Effect in Second Embodiment In the above configuration, the digital portable telephone with camera MS
3 is a target composed of five square-shaped targets 107A to 107E substantially at the center of the gesture recognition display area 106 on the gesture recognition screen 100 when the main control unit 250 activates the cyber gesture program 180. The units 107 are displayed in a row.

The digital mobile phone with camera MS
3 recognizes the movement in the left and right direction when the user's hand is moved, as in the first embodiment, and overlaps the pointer 108 on each of the targets 107A to 107E to match the movement of the user's hand. By moving the hand and displaying it in order, the user can confirm the trajectory of the hand movement and the recognition result.

As described above, the camera-equipped digital mobile phone MS3 feeds back and learns the trajectory of the hand movement and the recognition result when the hand movement can be recognized, and thereby the camera-equipped digital mobile phone MS3. M
Since it is possible to prompt the user to perform a hand movement that is easy to recognize in S3, it is possible to reduce the time until a command is input by the user.

According to the above configuration, the camera-equipped digital mobile phone MS3 recognizes the user's hand movement captured by the CCD camera 216 and, as a result, displays the trajectory of the hand movement and the recognition result as the targets 107A to 107E and the pointer 108.
By displaying the animation in an easy-to-understand manner using, the trajectory of the hand movement and the recognition result can be fed back to the user for learning.

(3) Other Embodiments In the first and second embodiments described above, the notebook personal computer 1 and the camera-equipped digital mobile phone MS3
The target 107 </ b> A- 1 </ b> A displays the trajectory of the left-right movement and the recognition result of the user's hand via visual feedback screens 191-193 as recognition process images.
07E, the pointer 108 is displayed while being moved in accordance with the movement of the user's hand, but the present invention is not limited to this. You may make it blink in order according to movement.

In the first and second embodiments described above, a case has been described in which a user's operation is recognized as a recognition target. However, the present invention is not limited to this. You may make it recognize the operation | movement of various other recognition objects, such as an animal.

Further, in the above-described first and second embodiments, the CP as the moving direction recognizing means and the control means is used.
U50 and 250 are hard disk or RO of HDD67.
Cyber gesture program 1 pre-stored in M
By displaying the animation while moving the pointer 108 on each of the targets 107A to 107E based on the position of the pointer 108 in accordance with the movement of the user's hand,
Although the case where the trajectory of the movement of the user's hand and the recognition result are notified to the user has been described, the present invention is not limited to this, and the program storage medium storing the cyber gesture program 180 is stored in the notebook computer 1 and By installing it in the digital mobile phone with camera MS3, the locus of the user's hand movement and the recognition result may be notified by animation display.

As described above, the cyber gesture program 180 for executing the series of processes described above is installed in the notebook personal computer 1 and the digital mobile phone with camera MS3, and can be executed in the notebook personal computer 1 and the digital mobile phone with camera MS3. Examples of the program storage medium used for setting the state include a floppy disk, a CD-ROM (Compac
t Disc-Read Only Memory), DVD (Digital Versatai
le Disc), a semiconductor memory or a magnetic disk in which the cyber gesture program 180 is temporarily or permanently stored. As a means for storing the cyber gesture program 180 in these program storage media, a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting may be used, and various communication interfaces such as a router and a modem may be used. Alternatively, the information may be stored via the Internet.

Further, in the first and second embodiments, a case has been described in which the information processing apparatus of the present invention is applied to the notebook personal computer 1 and the digital mobile phone with camera MS3. MS1
And other various information processing devices such as the MS2.

[0181]

As described above, according to the present invention, the moving direction of the recognition target in the image is recognized based on the image obtained by imaging the recognition target by the imaging means, and the recognized recognition target is recognized. By generating a recognition process image representing the trajectory of the movement direction and displaying it on a predetermined display means, it is possible to feed back and learn how the recognition target operation is recognized to the user.

[Brief description of the drawings]

FIG. 1 is a schematic perspective view showing an overall configuration of a notebook personal computer according to a first embodiment of the present invention.

FIG. 2 is a schematic diagram illustrating a configuration of a left side surface of a main body.

FIG. 3 is a schematic diagram illustrating a configuration of a rear side surface and a bottom surface of a main body.

FIG. 4 is a block diagram showing a circuit configuration of the notebook personal computer.

FIG. 5 is a flowchart illustrating a gesture recognition processing procedure.

FIG. 6 is a schematic diagram illustrating a gesture recognition screen displayed over the active window screen.

FIG. 7 is a schematic diagram illustrating a configuration of a gesture recognition screen.

FIG. 8 is a schematic diagram illustrating a configuration of a target.

FIG. 9 is a flowchart illustrating an acquisition processing procedure regarding hand position information.

FIG. 10 is a schematic diagram illustrating a color region represented on a YUV chromaticity space.

FIG. 11 is a schematic diagram illustrating a gesture recognition screen in a search state.

FIG. 12 is a schematic diagram illustrating a gesture recognition screen on which a pointer and a palm area recognition frame are overlapped and displayed.

FIG. 13 is a flowchart illustrating a determination processing procedure regarding a gesture operation.

FIG. 14 is a schematic diagram for explaining calculation of a fingertip moving distance;

FIG. 15 is a schematic diagram used to explain a flow of software processing;

FIG. 16 is a schematic diagram illustrating a visual feedback screen.

FIG. 17 is a schematic diagram illustrating an overall configuration of a network system according to a second embodiment.

FIG. 18 is a schematic perspective view showing an external configuration of a digital mobile phone with a camera.

FIG. 19 is a schematic perspective view showing the display unit when the camera unit is rotated.

FIG. 20 is a block diagram illustrating a circuit configuration of a digital mobile phone with a camera.

[Explanation of symbols]

1. Notebook type personal computer, 2, 2
13 ... body, 3, 212 ... display unit, 4, 219 ...
Operation keys, 8, 216 ... CCD camera, 10, 218
...... liquid crystal display, 11 ... imaging part, 24, 222
…… Jog dial, 50 …… CPU, 53 …… RA
M, 100: gesture recognition screen, 107: target unit, 108: pointer, 109: palm area recognition frame, 120: locus display frame, 121: direction display frame, 200: network system, MS1, MS2
…… Personal digital assistant, MS3, MS4 …… Digital mobile phone with camera, CS1-CS4 …… Base station, INW…
.. Public telephone network, 214 antenna 223 memory stick 250 main control unit 251 power supply circuit unit 252 operation input control unit 253 image encoder 254 camera interface unit , 255 ...
... LCD controller, 256 ... Image decoder, 257 ...
Demultiplexer, 258 Modulator / demodulator, 259 Audio codec, 260 Main bus, 261 Synchronous bus, 262 Recording / reproducing unit.

 ──────────────────────────────────────────────────続 き Continuation of the front page (72) Inventor Keigo Ihara 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation (72) Inventor Takahiko Sueyoshi 6-35-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony (72) Inventor Yoshihiro Yamaguchi 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation Inside (72) Inventor Shinichiro Gomi 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation F Terms (reference) 5C082 AA01 AA21 AA22 AA24 AA27 AA37 BA02 BA12 BA27 BD02 CA55 CA62 CB06 MM09

Claims (4)

[Claims] 1. A moving direction recognizing means for recognizing a moving direction of the recognition target based on an image obtained by imaging the recognition target by an imaging means, and a movement of the recognition target recognized by the moving direction recognizing means. A control unit for generating a recognition process image representing the trajectory of the direction and displaying the image on a predetermined display unit. 2. The apparatus according to claim 1, wherein the control means alternately displays, as the recognition process image, a plurality of marks of a predetermined shape sequentially arranged in the moving direction of the recognition target in the moving direction. Item 2. The information processing device according to item 1. 3. A moving direction recognition processing step of recognizing a moving direction of the recognition target based on an image obtained by imaging the recognition target by an imaging unit; and the recognition target recognized in the moving direction recognition processing step. A recognition process image generating process step of generating a recognition process image representing a trajectory of the moving direction of the moving object; and a display process step of displaying the recognition process image generated in the recognition process image generating process step on predetermined display means. A motion recognition process display method characterized by the following steps. 4. A moving direction recognition processing step of recognizing a moving direction of the recognition target based on an image obtained by imaging the recognition target by an imaging unit; and the recognition target recognized in the moving direction recognition processing step. A recognition process image generating process step of generating a recognition process image representing a trajectory of the moving direction of the moving object; and a display process step of displaying the recognition process image generated in the recognition process image generating process step on predetermined display means. A program storage medium for causing an information processing device to execute a program characterized by the following.