JP2006154898A - Portable terminal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a novel character input mode without adding special hardware in a cellphone terminal equipped with a camera. <P>SOLUTION: Photographing is carried out while moving a terminal body to describe a character by the camera 33 mounted to the cellphone terminal 100. A motion vector showing the motion of a photographed imaged body is detected by a motion vector detecting part 211, and the distributed state of the motion vector in a moving image is detected by a motion vector distribution detecting part 213. Based on the direction, size, distributed state, and the like of the detected motion vector, a dictionary database 6A is retrieved to execute character recognition. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an imaging apparatus such as a CCD camera that can capture a moving image, and a portable terminal that includes an input unit for inputting characters and control codes.

  Since a portable information terminal device including a mobile phone is small and excellent in portability, a large number of keys such as a personal computer cannot be provided in an input unit. Therefore, a small number of keys (for example, 15 phones) It is devised to input with the front and back keys, joystick keys, etc.). A so-called PDA (personal digital assistant) is also known that uses handwritten character recognition by pen input from a touch panel.

Japanese Patent Application Laid-Open No. 2004-133867 discloses a line sensor arranged at the bottom of a mobile phone to perform character input by performing character recognition. Patent Document 2 proposes a portable terminal that includes a touch sensor that is divided into four for each of 15 keys, and that enables character input by tracing the entire key like a touch panel. Japanese Patent Application Laid-Open No. 2004-228561 proposes a mobile terminal equipped with a motion sensor, and the terminal itself is moved back and forth and left and right to input characters and the like.
However, adding special hardware such as a line sensor or a touch sensor for character input leads to an increase in the volume of the mobile terminal and an increase in the price of the mobile phone terminal.

  Patent Document 4 discloses a technique for recognizing a user's gesture and switching screens using a camera attached to the mobile phone terminal. However, the technique of this document only allows screen switching and the like, and does not enable character input or the like.

JP 2003-199163 A ([0011] column and others) JP 2001-333166 A ([0016] column etc.) JP 2000-78262 A ([0009] to [0017] columns and others) JP 2000-83302 A ([0157] to [0181] columns and others)

  An object of the present invention is to provide a portable terminal capable of providing a new character input mode without adding special hardware.

  The portable terminal according to the present invention is a portable terminal having a function of inputting input information, and is imaged while moving the main body, an imaging unit that is provided integrally with the main body and captures a moving image, and moving the main body. A motion vector detection unit for detecting a motion vector indicating the motion of the imaging target in the video, a motion vector distribution detection unit for detecting a distribution state of the motion vector in the video, and a plurality of the motion vectors. A dictionary database that stores characters in association with each size and direction, and the distribution state, and corresponds to the size and direction detected by the motion vector detection unit and the motion vector distribution detection unit, and the distribution state. And a reading unit for reading out characters from the dictionary database.

  According to the present invention, a new character input mode can be provided in a portable terminal without adding special hardware.

Next, embodiments of the present invention will be described in detail with reference to the drawings. In addition, in the following explanation, an example of inputting characters such as numbers and alphabets is mainly explained. However, control codes such as figures, symbols, and mail sending commands are also considered as "characters" and are subject to input. Of course you can.
FIG. 1 is a block diagram showing a circuit configuration of a cellular phone terminal according to an embodiment of the present invention. The cellular phone terminal 100 includes a wireless unit 1, a baseband unit 2, an input / output unit 3, a power supply unit 4, and a vibrator 5. The mobile phone terminal 100 can be connected to a memory card 6 for storing the dictionary database 6A and the like according to the present embodiment.

  In the figure, a radio frequency signal arriving from a base station (not shown) via a radio channel is received by an antenna 11 and then input to a receiving circuit (RX) 13 via an antenna duplexer (DUP) 12. The reception circuit 13 includes a high frequency amplifier, a frequency converter, and a demodulator. The radio frequency signal is amplified by a low noise amplifier and then mixed with a reception local oscillation signal generated from a frequency synthesizer (SYN) 14 in a frequency converter to generate a reception intermediate frequency signal or a reception baseband signal. The output signal is digitally demodulated by a demodulator. As the demodulation method, for example, an orthogonal demodulation method corresponding to the QPSK method and a spectrum despreading method using a spreading code are used. The reception local oscillation signal frequency generated from the frequency synthesizer 14 is instructed from the main control unit 21 provided in the baseband unit 2.

  The demodulated signal output from the demodulator is input to the baseband unit 2. The baseband unit 2 includes a main control unit 21, a demultiplexing unit 22, an audio code decoding unit (hereinafter referred to as an audio codec) 23, an image processing unit 24, an LCD control unit 25, and a memory unit 26. I have. The memory unit 26 is a moving image data storage unit 261 that stores moving image data obtained by the camera 33, a moving vector data storage unit 262 that stores detected moving vectors, and a reference moving image that stores detected reference moving vectors. A vector storage unit 263, a character recognition result storage unit 264 that stores character recognition results, a voice data storage unit 265 that stores voice data, and the like are provided. The motion vector and the reference motion vector will be described later.

  The demodulated signal is identified by the main control unit 21 as control information or image information. If it is image information, it is supplied to the demultiplexing unit 22 where it is separated into audio data and image data. Then, the audio data is supplied to the audio codec 23 where it is decoded and the reproduced audio signal is output from the speaker 32 of the input / output unit 3 as a loud voice. On the other hand, the image data is supplied to the image processing unit 24 where the image decoding process is performed, and the image signal reproduced thereby is supplied to the LCD 34 of the input / output unit 3 via the LCD control unit 25 and displayed.

  Also, when reproducing and displaying audio data and image data stored in the memory unit 26, these data are input to the audio codec 23 and the image processing unit 24, respectively. The audio data is decoded by the audio codec 23 and then output from the speaker 32. The image data is decoded by the image processing unit 24 and then supplied to the LCD 34 via the LCD control unit 25 and displayed.

On the other hand, the user's transmitted voice signal output from the microphone 31 of the input / output unit 3 is input to the audio codec 23 of the baseband unit 2, and is encoded here and then input to the demultiplexing unit 22. The image signal output from the camera 33 is input to the image processing unit 24 of the baseband unit 2, and after being subjected to image encoding processing, is input to the demultiplexing unit 22. The demultiplexing unit 22 multiplexes the encoded audio data and image data in a predetermined format, and the multiplexed transmission data is sent from the main control unit 21 to the transmission circuit (TX) 15 of the radio unit 1. Entered.
Note that the camera 33 may be either one capable of frame photographing or one capable of field photographing. For example, 30 frames are shot in one second, and each frame forms one image. Field shooting is a shooting in which a screen can be divided according to odd lines and even lines ( Interlace shooting), and one frame is composed of two fields. In the following description, it is assumed that the camera in FIG. 1 performs frame shooting.

  The transmission circuit 15 includes a modulator, a frequency converter, and a transmission power amplifier. The transmission data is digitally modulated by the modulator, then mixed with the transmission local oscillation signal generated from the frequency synthesizer 14 by the frequency converter and frequency-converted to a radio frequency signal. As a modulation method, a QPSK method and a spread spectrum method using a spread code are used. The generated transmission radio frequency signal is amplified to a predetermined transmission level by a transmission power amplifier, then supplied to the antenna 11 via the antenna duplexer 12, and transmitted from the antenna 11 to a base station (not shown). Is done.

  The power supply unit 4 includes a battery 41 such as a lithium ion battery, a charging circuit 42 for charging the battery 41 based on a commercial power output (AC 100 V), and a voltage generation circuit (PS) 43. It has been. The voltage generation circuit 43 is composed of, for example, a DC / DC converter, and generates a predetermined power supply voltage Vcc based on the output voltage of the battery 41.

  Since the main control unit 21 performs character input according to the present embodiment, as shown in FIG. 1, the motion vector detection unit 211, the motion vector narrowing processing unit 212, the motion vector distribution detection unit 214, the speech recognition unit 215, and A character recognition unit 216 is included. In the present embodiment, as shown in FIG. 2, the mobile phone terminal 100 is photographed by the camera 33 while moving the mobile phone terminal 100 so as to draw a character to be input (in FIG. 2, the user's left hand finger F). Input characters. For example, in order to input the number “6”, as shown in FIG. 2, after the character input activation key 35A is pressed to activate the character input mode by the camera 33, the camera 33 is directed toward the finger F. While photographing F, the cellular phone terminal 100 is moved to draw “6” while pressing the character input key 35B. The character input key 35B starts to be pressed at the start point of the locus for drawing “6” and is released at the end point. Thus, the character input can be performed by detecting the motion vector indicating the relative motion of the finger F taken with respect to the camera 33 while the character input key 35B is pressed by the motion vector detection unit 211. That is, the character input key 35B functions as means for inputting information indicating the start and end of one image constituting the character. Characters having two or more strokes can be input by releasing the character input key 35B when the characters of one stroke are completed and pressing the character input key 35B again when inputting the next one stroke.

Note that the imaging target such as the finger F is preferably a stationary object, that is, an object that does not move from a predetermined position while the camera 33 is photographing. Other than the finger F in the example of FIG. 2, any stationary object that can be photographed and that can clearly detect a motion vector may be used.
Also, instead of holding down the character input key 35B during the input of one stroke, the start and end of one stroke are input by pressing the character input key 35B once at the start and end of one stroke. May be. Alternatively, the start and end of one stroke of a character may be input by voice input from the microphone 31.

  The motion vector detection unit 211 has a function of calculating the size and direction of a motion vector indicating the motion of a pixel in a predetermined region of each frame constituting a motion image acquired by the camera 33. In this embodiment, a motion vector refers to a vector representing the direction and distance in which a moving image has moved between, for example, the previous and subsequent frames as reference frames and the current frame. When conforming to the MPEG (Motion Picture Encoding Group) standard, a reference frame is set as an I picture, and an I picture is set at a predetermined setting interval. Then, using this I picture as a reference, the motion vectors of the previous and subsequent frames (B picture, P picture) are calculated. In this embodiment, since the purpose is not to compress and encode an image, the motion vector is only used as a target for detecting the movement of the finger F with respect to the camera 33, and is required to be used for the compression encoding. is not. However, after compression encoding, it is possible to detect a motion vector based on the compressed image.

As an example, as illustrated in FIG. 3, the motion vector detection unit 211 converts one frame of horizontal X pixels and vertical Y pixels (for example, X = 360, Y = 240) into horizontal XB pixels × vertical YB pixels (for example, XB = YB = 4). Then, it is configured to calculate VN (= X / XB × Y / YB) motion vectors for each macroblock. For example, as shown in FIG. 3, consider a case where there are a current frame N + T and a past frame N, and an image that was present in the macroblock M in the frame N has moved to the macroblock S in the frame N + T. . At this time, the motion vector Vm in the macroblock S can be calculated from data indicating the difference in the coordinate values of the macroblocks M and S and other differences in position. For other macroblocks, motion vectors can be calculated similarly.
Note that the macroblock is not limited to a rectangle, and may be, for example, a circle, an ellipse, a triangle, or a polygon having five or more corners. The motion vector can also be calculated using a forward prediction calculation and a backward prediction calculation over a plurality of frames.

  Alternatively, a specific region in one frame can be defined as a motion vector extraction region, and a representative motion vector can be obtained for each motion vector extraction region. For example, as shown in FIG. 4, the motion vector extraction regions define first to fourth motion vector extraction regions 302 to 305 having substantially the same shape and size, with the screen center interposed therebetween, and each region. The representative motion vectors 302A to 305A can be obtained for each. In some cases, as illustrated in FIG. 5, the size and shape of the individual extraction regions 302 ′ to 305 ′ may be different from each other. It is preferable not to set an extraction region in a region where a motion vector is unlikely to occur and noise is likely to enter, such as a corner portion of a screen.

  The motion vector detection unit 211 detects not only the direction of the motion vector but also the size thereof. According to only the direction of the motion vector, for example, it becomes difficult to distinguish the numbers “0” and “6”, and the possibility of erroneous recognition and the like increases. The magnitude of the motion vector is data corresponding to the moving speed of the mobile phone terminal 100. In the case of drawing “0” and the case of drawing “6”, the loop drawn by the latter is smaller and the loop closes earlier, so that there is a difference in the moving speed of the loop. Accordingly, by detecting the moving speed, that is, the size of the motion vector, for example, the similar characters “0” and “6” can be easily identified.

The motion vector narrowing processing unit 212 has a function of executing a predetermined data removal process on the motion vector detected by the motion vector detecting unit 211 to acquire a reference motion vector obtained by narrowing motion vector data. The data removal process to be executed is, for example,
(1) Processing for removing disturbance data by absorbing variations in motion vectors in one frame calculated by the motion vector detection unit 211;
(2) Processing to remove disturbance data by performing frame thinning,
(3) Processing for detecting an edge of an object image in a moving image and removing other than the moving vector in the vicinity of the edge, or (4) Processing of a combination thereof.

  For example, when a character input according to the present embodiment is performed in a moving vehicle or the like, a building, a car, or the like may move in the background, and this may appear on the screen. Such movement of the object appears as a variation with respect to the entire motion vector on the frame. In such a case, by calculating the total average of VN motion vectors or the correlation coefficient of a plurality of representative vectors, variations can be absorbed and the influence of such an object image can be removed. Alternatively, the standard deviation σ of VN motion vectors is obtained, and only motion vectors within the range of a predetermined value KI · σ (KI is a positive value coefficient) are left, and motion vectors outside this range are removed. Depending on, it is possible to narrow down.

  Alternatively, the number of motion vectors can be narrowed down by thinning out the number of frames AA included in the captured moving image to a smaller number of frames AP. The method of thinning the number of frames AA to the AP is, for example, (1) thinning a fixed number of characters between the start point and end point of one stroke of a character, regardless of the length of time, (2) frames after thinning The time interval between them is a predetermined period TT, (3) the period TT is an unequal interval, (4) the correlation of each motion vector in adjacent frames is calculated, and a high correlation greater than a predetermined threshold TSH is obtained. It is conceivable to thin out the frame that has it.

  Alternatively, it is possible to estimate the movement of the small object in the moving frame from the correlation between the frames and remove it. In this way, the character recognition accuracy in the character recognition unit 216 can be improved by performing the data removal processing that is determined to be ineffective for estimating the motion vector based on the relative movement of the finger F.

  The motion vector distribution detection unit 213 has a function of detecting the distribution state of each motion vector on each frame after being detected and narrowed down in each frame. For example, as shown in FIG. 6, when the finger F moves on the screen due to the mobile phone terminal 100 being moved so as to draw characters in the frames 1101 to 1106, the motion vectors are also displayed in the frames 1101 to 1106. 2, the distribution range moves along the moving direction of the finger F. For example, by calculating a vector 1108 indicating the moving direction of the distribution as shown in FIG. 6, the distribution state of the motion vector in each frame can be detected.

  The voice recognition unit 215 has a function of recognizing voice information input from the microphone 31 as auxiliary information for character recognition using a motion vector obtained by photographing the finger F or the like while moving the mobile phone terminal 100. Various types of speech can be considered. For example, (1) speech indicating the input character itself, (2) speech indicating the shape of the input character, or the like can be input. As an example of the former, for example, in order to identify characters having similar shapes, for example, “0” and “6”, the reading can be inputted from the microphone 31 by voice. As an example of the latter, for example, when inputting “+” and “T” having similar shapes, the mobile phone terminal 100 is moved and a voice indicating the intersection (eg, “cross”) is displayed at the intersection of each image. By inputting from the microphone 31, both can be identified.

The character recognition unit 216 detects the direction and size of the motion vector after being detected by the motion vector detection unit 211 and subjected to the narrowing processing by the motion vector narrowing processing unit, and the distribution state of the motion vector detected by the motion vector distribution detection unit 213 Is retrieved from the dictionary database 6A, and the retrieval result is read out.
The dictionary database 6A stores data relating to the direction and size of the motion vector and the distribution state, and various characters in association with each other. The learning process of the dictionary database 6A can be performed by the same procedure as the character input and recognition, but it is also possible to use data input using other imaging devices having the same resolution and the same optical characteristics. It is. Further, the learning process method for the dictionary database 6A is not limited as long as it is possible to input motion vector information without using an imaging device.

  The imaging device used at the time of learning may have a resolution and optical characteristics different from those of the camera 33. Further, at the time of learning, a motion vector detection method (such as setting of a macro block and a motion vector extraction region) may be different from that in character input and recognition. In this case, the motion vector narrowing-down processing unit 212 adjusts the number M2 of motion vectors detected by the motion vector detection unit 211 to the number M1 of motion vectors stored as data in the dictionary database 6A (for example, Characters can be recognized based on the motion vectors after the adjustment to M3 motion vectors smaller than M1 and M2.

  Next, a procedure for inputting characters by the mobile phone terminal according to the present embodiment will be described with reference to the flowchart of FIG. Here, a case where a character “+” consisting of two strokes is input will be described as an example. When starting character input, the user first presses the character input activation key 35A (S11). An object to be imaged such as a finger F is placed in front of the lens of the camera 33, the camera 33 is pointed in the direction of the finger F, and the character input key 35B is started to be pressed (S12). Move so as to draw one horizontal line (S13). In addition, while the mobile phone terminal 100 is moving, voice is input from the microphone 31 as necessary (S14), but voice input is not necessary when drawing this horizontal line.

  When the movement in the horizontal direction is finished, the character input key 35B is released (S15). A motion vector indicating the relative movement of the finger F taken with respect to the camera 33 while the character input key 35B is pressed is detected by the motion vector detection unit 211 (S16). After the motion vector narrowing processing is performed in the motion vector narrowing processing unit 212 (S17), the motion vector distribution detection unit 213 detects the distribution state of these motion vectors (S18). When inputting characters having two or more strokes, the procedure from S12 to S18 is repeated until the input of all the images constituting the character to be input is completed (S19). When a vertical line, which is the second stroke of the character “+”, is input, voice (“cross” or the like) indicating an intersection point is input to the microphone 31 while the mobile phone 100 is moving in the vertical direction (from top to bottom). The user speaks toward The character recognizing unit 216 recognizes the input character based on the contents of the voice, the timing information issued, and the motion vector information (S20).

  When a character “T” similar to “+” is input, a voice (“cross” or the like) indicating an intersection is not given at the beginning of the movement of the mobile phone terminal 100 in the vertical direction, but during the movement. Speak. In this manner, similar characters can be identified by varying the timing of utterance depending on the input characters. It is also possible to input a voice (“plus”, “tee”, etc.) indicating the input character reading itself.

  Although the embodiments of the invention have been described above, the present invention is not limited to these embodiments, and various modifications and additions can be made without departing from the spirit of the invention. For example, in the above embodiment, a moving image is picked up by one camera 33 having one lens, but a plurality of time-synchronous or asynchronous moving images obtained by a camera having a plurality of lenses are selected. Alternatively, a plurality of series of images may be superimposed and used as a moving image. Furthermore, when superimposing images, they may be superimposed after multiplying by a separately determined weighting coefficient.

  In the above embodiment, the motion vector information (direction, size, and distribution state) is stored in the dictionary database 6A in association with the characters. Instead, the mobile phone terminal calculated from the motion vector information. 100 moving directions and speeds may be stored in the dictionary database 6A. The moving direction of the mobile phone terminal 100 is opposite to the direction of the motion vector. It is also possible to store information related to speech input at the time of character input in the dictionary database 6A and use it at the time of character recognition.

  In the above embodiment, information for identifying similar characters from each other is input by the microphone 31. However, the present invention is not limited to this, and various information may be input. For example, information relating to the type of characters (English characters, hiragana, katakana, numbers, etc.) may be input by voice.

  When the character input mode according to the present embodiment is terminated, the key 35A is operated in the above example. Instead, for example, the lens of the camera 33 has a predetermined period of TE seconds (for example, about 2 to 5 seconds). ) When the character is continuously pressed, the character input mode of this embodiment may be terminated.

  Further, as shown in FIG. 8, in the flip-type portable information terminal device in which the flip 34 and the main body 37 can be opened and closed by the hinge portion 38, the surface of the flip 34 (the surface seen when the flip is closed). May be provided with the character input button 35B of the present embodiment.

In the above embodiment, character recognition is performed based on the direction and magnitude of the motion vector detected by the motion vector detection unit 211 and the distribution state of the motion vector detected by the motion vector distribution detection unit 213. However, the present invention is not limited to this. For example, character recognition may be executed based only on the distribution state. At this time, the direction or size of each motion vector may be grasped as a part of data constituting the “distribution state” and may be detected by the motion vector distribution detection unit 213.
In addition, the pixel size may be various sizes or formats in the present invention. For example, ITU-T H.261 CIF, MPEG-1 SIF, NTSC image, PAL image, HDTV image, Any image of a different size may be used.

It is a block diagram which shows the circuit structure of the mobile telephone terminal concerning embodiment of this invention. It is a conceptual diagram which shows a mode that the character input mode by this Embodiment is performed. An example of the operation of the motion vector detection unit 211 will be described. An example of a motion vector extraction region set in a frame is shown. An example of a motion vector extraction region set in a frame is shown. An example of the operation of the motion vector distribution detection unit 213 will be shown. It is a flowchart which shows the procedure of the character input mode by the mobile telephone terminal 100 of this Embodiment. One of the modifications of this Embodiment is shown.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Radio part, 2 ... Baseband part, 3 ... Input / output part, 4 ... Power supply part, 5 ... Vibrator, 6 ... Memory card, 11 ... Antenna, 12 ... Antenna duplexer (DUP), 13 ... Reception circuit (RX), 14 ... Frequency synthesizer (SYN), 15 ... Transmission circuit, 21 ... Main control unit, 22 ... Multiplexing Separation unit, 23 ... Audio codec, 24 ... Image processing unit, 25 ... LCD control unit, 26 ... Memory unit, 31 ... Microphone, 32 ... Speaker, 33 ... Camera 34 ... LCD, 35 ... keyboard, 41 ... battery, 42 ... charge circuit, 43 ... voltage generation circuit, 211 ... motion vector detection unit, 212 ... motion vector narrowing down Processing unit, 213 ... motion vector distribution detection unit section, 215 ... voice recognition unit, 216 ... character recognition unit.

Claims (10)

In a mobile terminal equipped with a function for inputting input information,
The main body,
An imaging unit that is provided integrally with the main body unit to capture a moving image;
A motion vector detection unit that detects a motion vector indicating a motion of the imaging target in a moving image captured while moving the main body;
A motion vector distribution detector for detecting a distribution state of the motion vector in the moving image;
A dictionary database that stores characters in association with the size and direction of each of the plurality of motion vectors and the distribution state;
A mobile terminal comprising: a reading unit that reads out from the dictionary database characters corresponding to the size and direction detected by the motion vector detection unit and the motion vector distribution detection unit and the distribution state.   The mobile terminal according to claim 1, further comprising: a voice identification unit that identifies voice, wherein the reading unit reads the size and direction and characters corresponding to the voice from the dictionary database. A one-stroke start / end input unit for inputting information indicating the start and end of drawing of a stroke of a character having two or more strokes;
The mobile terminal according to claim 1, wherein the motion vector detection unit operates between the start and end indicated by the one-stroke start / end input unit.   The mobile terminal according to claim 1, further comprising a motion vector narrowing processing unit that narrows down a plurality of motion vectors detected by the motion vector detection unit according to a predetermined narrowing criterion and outputs a reference motion vector.   The mobile terminal according to claim 4, wherein the motion vector narrowing-down processing unit executes processing for absorbing variations in motion vectors detected by the motion vector detection unit.   The mobile terminal according to claim 4, wherein the motion vector narrowing processing unit performs thinning of the images.   The mobile terminal according to claim 4, wherein the motion vector narrowing-down processing unit detects an edge of an object image in the motion image and executes a process of removing motion vectors other than the motion vector near the edge.   The portable terminal according to claim 1, further comprising means for forcibly terminating the end of the character input mode when it is determined that the black image continues for a predetermined time or longer in the moving image. A portable main body with a function for inputting information;
An imaging unit that is provided integrally with the main body unit and captures a moving image;
A motion vector detection unit that sequentially detects a motion vector indicating a motion of an imaging target in the moving image captured while moving the main body, and detects a distribution state of the motion vector in the moving image;
A dictionary database that stores distribution states of a plurality of motion vectors previously associated with each character symbol;
A mobile terminal comprising: a search unit that searches the dictionary database for a character symbol associated with a distribution state most similar to the distribution state detected by the motion vector distribution detection unit.   The motion vector distribution detection unit includes means for detecting at least one of the magnitude and direction of a motion vector as a feature amount, and means for outputting a set of a plurality of detected feature amounts as the distribution state. The mobile terminal according to claim 9, which is characterized by:

Images