JP4381282B2 - Mobile device - Google Patents

Description

    The present invention relates to an imaging apparatus such as a CCD camera that can capture a moving image, and a portable terminal that includes an input unit for inputting characters and control codes.

  Since a portable information terminal device including a mobile phone is small and excellent in portability, a large number of keys such as a personal computer cannot be provided in an input unit. Therefore, a small number of keys (for example, 15 phones) It is devised to input with the front and back keys, joystick keys, etc.). A so-called PDA (personal digital assistant) is also known that uses handwritten character recognition by pen input from a touch panel.

Japanese Patent Application Laid-Open No. 2004-133867 discloses a line sensor arranged at the bottom of a mobile phone to perform character input by performing character recognition. Patent Document 2 proposes a portable terminal that includes a touch sensor that is divided into four for each of 15 keys, and that enables character input by tracing the entire key like a touch panel. Japanese Patent Application Laid-Open No. 2004-228561 proposes a mobile terminal equipped with a motion sensor, and the terminal itself is moved back and forth and left and right to input characters and the like.
However, adding special hardware such as a line sensor or a touch sensor for character input leads to an increase in the volume of the mobile terminal and an increase in the price of the mobile phone terminal.
JP 2003-199163 A ([0011] column and others) JP 2001-333166 A ([0016] column etc.) JP 2000-78262 A ([0009] to [0017] columns and others)

  An object of the present invention is to provide a portable terminal capable of providing a new character input mode without adding special hardware.

A portable terminal according to the present invention is a portable terminal having a function of inputting input information, an imaging unit that captures a moving image, and a black image that determines whether or not the image included in the moving image is a black image Detected by a determination unit, a black image timing detection unit that detects a generation timing at which the black image is generated, a dictionary database that stores the generation timing and the input information in association with each other, and the black image timing detection unit A read unit that reads out the input information corresponding to the generation timing from the dictionary database , and the black image determination unit calculates a motion vector that indicates a motion of a black pixel in a predetermined region on the image. An image including the motion vector is determined as the black image when the motion indicated by the motion vector is equal to or less than a predetermined amount. To.

  According to the present invention, a new character input mode can be provided in a portable terminal without adding special hardware.

Next, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a circuit configuration of a cellular phone terminal according to an embodiment of the present invention. The cellular phone terminal 100 includes a wireless unit 1, a baseband unit 2, an input / output unit 3, a power supply unit 4, and a vibrator 5. The mobile phone terminal 100 can be connected to a memory card 6 for storing the dictionary database 6A and the like according to the present embodiment.

  In the figure, a radio frequency signal arriving from a base station (not shown) via a radio channel is received by an antenna 11 and then input to a receiving circuit (RX) 13 via an antenna duplexer (DUP) 12. The reception circuit 13 includes a high frequency amplifier, a frequency converter, and a demodulator. The radio frequency signal is amplified by a low noise amplifier and then mixed with a reception local oscillation signal generated from a frequency synthesizer (SYN) 14 in a frequency converter to generate a reception intermediate frequency signal or a reception baseband signal. The output signal is digitally demodulated by a demodulator. As the demodulation method, for example, an orthogonal demodulation method corresponding to the QPSK method and a spectrum despreading method using a spreading code are used. The reception local oscillation signal frequency generated from the frequency synthesizer 14 is instructed from the main control unit 21 provided in the baseband unit 2.

  The demodulated signal output from the demodulator is input to the baseband unit 2. The baseband unit 2 includes a main control unit 21, a demultiplexing unit 22, an audio code decoding unit (hereinafter referred to as an audio codec) 23, an image processing unit 24, an LCD control unit 25, and a memory unit 26. I have. The memory unit 26 is a moving image data storage unit 261 that stores moving image data obtained by the camera 33, a moving vector data storage unit 262 that stores detected moving vectors, and a reference moving image that stores detected reference moving vectors. A vector storage unit 263, a character recognition result storage unit 264 that stores character recognition results, a voice data storage unit 265 that stores voice data, and the like are provided. The motion vector and the reference motion vector will be described later.

  The demodulated signal is identified by the main control unit 21 as control information or image information. If it is image information, it is supplied to the demultiplexing unit 22 where it is separated into audio data and image data. Then, the audio data is supplied to the audio codec 23 where it is decoded, and the reproduced audio signal is amplified and output from the speaker 32 of the input / output unit 3. On the other hand, the image data is supplied to the image processing unit 24 where the image decoding process is performed, and the image signal reproduced thereby is supplied to the LCD 34 of the input / output unit 3 via the LCD control unit 25 and displayed.

  Also, when reproducing and displaying audio data and image data stored in the memory unit 26, these data are input to the audio codec 23 and the image processing unit 24, respectively. The audio data is decoded by the audio codec 23 and then output from the speaker 32. The image data is decoded by the image processing unit 24 and then supplied to the LCD 34 via the LCD control unit 25 and displayed.

On the other hand, the user's transmitted voice signal output from the microphone 31 of the input / output unit 3 is input to the audio codec 23 of the baseband unit 2, and is encoded here and then input to the demultiplexing unit 22. The image signal output from the camera 33 is input to the image processing unit 24 of the baseband unit 2, and after being subjected to image encoding processing, is input to the demultiplexing unit 22. The demultiplexing unit 22 multiplexes the encoded audio data and image data in a predetermined format, and the multiplexed transmission data is sent from the main control unit 21 to the transmission circuit (TX) 15 of the radio unit 1. Entered.
Note that the camera 33 may be either one capable of frame photographing or one capable of field photographing. For example, 30 frames are shot in one second, and each frame forms one image. Field shooting is a shooting in which a screen can be divided according to odd lines and even lines ( Interlace shooting), and one frame is composed of two fields. In the following description, it is assumed that the camera in FIG. 1 performs frame shooting.

  The transmission circuit 15 includes a modulator, a frequency converter, and a transmission power amplifier. The transmission data is digitally modulated by the modulator, then mixed with the transmission local oscillation signal generated from the frequency synthesizer 14 by the frequency converter and frequency-converted to a radio frequency signal. As a modulation method, a QPSK method and a spread spectrum method using a spread code are used. The generated transmission radio frequency signal is amplified to a predetermined transmission level by a transmission power amplifier, then supplied to the antenna 11 via the antenna duplexer 12, and transmitted from the antenna 11 to a base station (not shown). Is done.

  The power supply unit 4 includes a battery 41 such as a lithium ion battery, a charging circuit 42 for charging the battery 41 based on a commercial power output (AC 100 V), and a voltage generation circuit (PS) 43. It has been. The voltage generation circuit 43 is composed of, for example, a DC / DC converter, and generates a predetermined power supply voltage Vcc based on the output voltage of the battery 41.

  The main control unit 21 includes a black image determination unit 211, a black image timing detection unit 212, a character recognition unit 213, and a timer 214 as shown in FIG. 1 in order to perform character input according to the present embodiment. In this embodiment, as shown in FIG. 2A, the user can select a character input mode for inputting characters by hitting (hiding) the lens of the camera 33 with a finger (F) or the like at a predetermined timing. it can. That is, for example, the user hits the lens of the camera 33 at a timing corresponding to a character or control code to be input (for example, a timing specified in the correspondence table shown in FIG. 2B) along a Morse code sequence. In FIG. 2B, “•” means that the lens of the camera 33 is hit with a finger and released immediately, and “−” means that the lens of the camera 33 is pressed for a longer timing than “•”. .

  For example, the input of the character “A” can be performed by inputting “-” after inputting “·” and then inputting “−”. The correspondence table in FIG. 2B is merely an example, and it is possible to further increase the timing information and increase the types of characters that can be input. This correspondence table can be input from the mobile phone terminal 100, or can be input by another device (not shown) and loaded into the mobile phone terminal 100. Further, the Morse code sequence is merely an example. For example, a rhythm of a specific music is stored in the dictionary database 6A in association with a character, and the lens is struck with a finger at a rhythm corresponding to the character to be input. It may be.

  The main control unit 21 detects the timing of hitting the camera 33 and recognizes the character that the user is trying to input. This method is effective as a means for substituting for character input by the keyboard 35 or the like or a means for complementing it. The relationship between the timing of hitting the lens and the corresponding characters (as shown in the table of FIG. 2B as an example) is stored as a dictionary database 6A in the memory card 6 or the like. The timing when the lens is struck is detected. Then, detection timing information that is the same as or similar to this detection timing is extracted from the dictionary database 6A. In order to simplify the recognition process, for example, the features of the detection timing can be vector quantized, averaged, and recognized for each representative codebook. Characters can be input by reading out characters and the like corresponding to the detected detection timing information from the dictionary database 6A.

  The above character input mode can be obtained by, for example, pressing the character input start / end key 35A shown in FIG. 2A once (when combined with other functions, a long press (the length TI for pressing the normal key). Can be activated or terminated (by pressing the key for a predetermined period TIL seconds longer than 2 seconds) or by multi-clicking (by clicking the key multiple times during the predetermined period TS seconds) it can.

  When the lens of the camera 33 is struck and covered with a finger, an image captured by the camera 33 is an image whose brightness is all or part of a predetermined value or less (hereinafter, such an image is referred to as a “black image”). Conversely, an image that is not a “black image” is hereinafter referred to as a “white image”). The black image determination unit 211 has a function of determining whether or not each frame included in the image signal input from the camera 33 is a black image. The specific configuration and operation of the black image determination unit 211 will be described later. In addition to or instead of the lightness determination, it may be determined whether the image is a black image by determining the saturation and the hue.

The black image timing detection unit 212 detects at what timing a frame determined to be a black image by the black image determination unit 211 is generated.
The character recognizing unit 213 reads a character or the like associated with timing information that is the same as or similar to the black image generation timing detected by the black image timing detecting unit 212 from the dictionary database 6A stored in the memory card 6. Have For the determination of identity or similarity, a recognition method using HMM (Hidden Markov Model) or a neural network can be used in addition to continuous DP (Dynamic Programming) matching. If there are a plurality of candidate characters, all of them may be displayed on the LCD 34 and may be selected by the user using the keyboard 35 or the like.
The timer 214 provides a clock signal used by the black image timing detection unit 212 to determine the aforementioned timing.

  The dictionary data (for example, data as shown in the table of FIG. 2B) of the dictionary database 6A is generated after the timing information is obtained by hitting the lens of the camera 33 as in the case of the character input. This can be done by inputting and storing the character desired to correspond to the keyboard 35. However, the information may be separately obtained from an external device and stored in the dictionary database 6A regardless of the components of the mobile phone terminal 100 such as the camera 33 and the keyboard 35.

As shown in FIG. 1, the black image determination unit 211 includes, for example, a motion vector calculation processing unit 2111, a motion vector narrowing calculation processing unit 2112, and a black image content rate calculation processing unit 2113.
The motion vector calculation processing unit 2111 has a function of calculating a motion vector indicating the motion of a pixel in a predetermined region of each frame of a moving image acquired by the camera 33. In this embodiment, a motion vector refers to a vector representing the direction and distance in which a moving image has moved between, for example, the previous and subsequent frames as reference frames and the current frame. When conforming to the MPEG (Motion Picture Encoding Group) standard, a reference frame is set as an I picture, and an I picture is set at a predetermined setting interval. Then, using this I picture as a reference, the motion vectors of the previous and subsequent frames (B picture, P picture) are calculated. In this embodiment, since the purpose is not to compress and encode an image, the motion vector is only used as a target for detecting the movement of the finger F, and is not required to be used for the compression encoding. However, after compression encoding, it is possible to determine whether or not the image is a black image based on the compressed image.

As an example, as shown in FIG. 3, the motion vector calculation processing unit 2111 converts one frame of horizontal X pixels and vertical Y pixels (for example, X = 360, Y = 240) into horizontal XB pixels × vertical YB pixels (for example, It is divided into macro blocks of XB = YB = 4). Then, it is configured to calculate VN (= X / XB × Y / YB) motion vectors for each macroblock. For example, as shown in FIG. 3, consider a case where there are a current frame N + T and a past frame N, and an image that was present in the macroblock M in the frame N has moved to the macroblock S in the frame N + T. . At this time, the motion vector Vm in the macroblock S can be calculated from data indicating the difference in the coordinate values of the macroblocks M and S and other differences in position. For other macroblocks, motion vectors can be calculated similarly.
Note that the macroblock is not limited to a rectangle, and may be, for example, a circle, an ellipse, a triangle, or a polygon having five or more corners. The motion vector can also be calculated using a forward prediction calculation and a backward prediction calculation over a plurality of frames.

  Alternatively, a specific region in one frame can be defined as a motion vector extraction region, and a representative motion vector can be obtained for each motion vector extraction region. It is preferable to determine in which region in one frame the motion vector extraction region is defined depending on the feature or tendency of the motion of the finger F to be detected. As an example, as shown in FIG. 4, first and second motion vector extraction regions 302 and 303 having substantially the same shape and size can be defined substantially symmetrically with respect to the center of the screen. In some cases, as shown in FIG. 5, the size and shape of the individual extraction regions 302 ', 303', and 304 'may be different from each other. It is preferable not to set the extraction area in an area where there is a high possibility that an image other than the finger F enters, such as a corner portion of the screen, and there is a low possibility that the image of the finger F is reflected.

The motion vector narrowing calculation processing unit 2112 has a function of executing a predetermined data removal process to acquire a reference motion vector obtained by narrowing down the calculated motion vector data. The data removal process to be executed is, for example,
(1) A process of removing disturbance data by absorbing variations in a motion vector in one frame calculated by the motion vector calculation processing unit 2111;
(2) Processing to remove disturbance data by performing frame thinning,
(3) Processing for detecting an edge of an object image in a moving image and removing other than the moving vector in the vicinity of the edge, or (4) Processing of a combination thereof.

  For example, when the character input according to the present embodiment is performed in a moving car or the like, an object unrelated to the finger F such as a building or a car may move in the background and be reflected on the screen. Such movement of the object appears as a variation with respect to the entire motion vector on the frame. In such a case, by calculating the total average of VN motion vectors or the correlation coefficient of a plurality of representative vectors, variations can be absorbed and the influence of such an object image can be removed. Alternatively, the standard deviation σ of VN motion vectors is obtained, and only motion vectors within the range of a predetermined value KI · σ (KI is a positive value coefficient) are left, and motion vectors outside this range are removed. Depending on, it is possible to narrow down.

  Alternatively, it is possible to estimate the movement of the small object in the moving frame from the correlation between the frames and remove it. In this way, the character recognition accuracy in the character recognition unit 213 can be improved by performing the data removal processing that is determined to be ineffective for estimating the motion vector based on the movement of the finger F.

  The black pixel content rate calculation processing unit 2113 is a pixel that is determined to be a black pixel from lightness, hue, saturation, or the like in one entire frame included in the moving image acquired by the camera 33 or in a specific region therein ( It has a function of calculating the content (ratio) of (black pixels).

  The black image determination unit 211 determines whether or not the target frame is a black image using only one or both of the calculation result of the motion vector calculation processing unit 2111 and the calculation result of the black pixel content rate calculation processing unit 2113. To do. When a black image can be detected with high accuracy only by the calculation result of the motion vector calculation processing 2111, only one of them can be operated, for example, the operation of the black pixel content rate calculation processing unit 2113 is stopped. The selection may be made by the user determining the background situation or the like, or the black image determination unit 211 may automatically make the selection.

  When only the calculation result of the motion vector calculation processing unit 2111 is used, the black image determination unit 211, for example, when the motion indicated by the reference motion vector output from the motion vector narrowing calculation processing unit 2112 is smaller than a predetermined value, the main control unit 21 Can determine a frame including the motion vector as a black image.

  When only the calculation result of the black pixel content rate calculation processing unit 2114 is used, the black image determination unit 2111 provides, for example, N determination regions in the rectangular region of the pixel X1 × Y1 at the center of the frame, In the case where the determination areas of M locations (M ≦ N) or more are black pixel content B% (for example, 80%), it can be determined that the frame is a black image. The determination region does not need to be the center of the screen, and may be provided asymmetrically on one end side of the screen depending on circumstances. The size and position of the determination region, the threshold values for M and B, and the like can be changed as appropriate based on detection accuracy and the like. For example, when only one frame of a black image is detected with a certain setting, it is suspected that the image to be determined as a black image is erroneously determined as a white image, so the size and position of the determination region, M, Appropriate determination is made by appropriately changing the threshold value of B or the like.

  In the case of such a determination method, even if the optical characteristics of the camera 33 change, the determination result is hardly affected. In addition, even if the number of pixels of the camera used when creating the dictionary database 6A is different from the number of pixels of the camera 33 used when inputting characters, the recognition accuracy is hardly affected.

  Next, the operation of the present embodiment will be described with reference to FIG. Here, images of the motion until the finger F gradually approaches the lens of the camera 33 and finally completely covers the lens are captured as frames 1101 to 1106 (the lens is completely covered by the finger F in the frame 1106). In this state, the movement of the finger F is detected as a motion vector Vm. Further, FIG. 6 shows an example in which the edge of the image of the finger F is detected in the motion vector narrowing calculation processing unit 2112 and motion vectors other than the motion vector near the edge are removed. By removing motion vectors other than those near the edges, the load on the black image determination unit 211 is reduced, and the determination speed can be increased. If the calculation load is not a problem, the edge detection operation may be stopped and the motion vector of the entire screen may be detected. Further, instead of edge detection, binarization processing may be performed on all pixels.

In this example, immediately before the finger F approaches the lens and covers the lens (frame 1105), the size of the motion vector decreases, and the position of the motion vector diffuses toward the outer periphery in the frame. Go. When this tendency is detected, it is determined that the finger F has covered the lens, and an image sequence of several frames with a small amount of motion vector immediately after that can be determined as a black image.
Note that the size of the motion vector also decreases when shooting a background with little motion, such as a single-color wall (it will be based on camera shake). No diffusion tendency occurs. Therefore, the case where the background does not move and the case where the finger F covers the lens can be discriminated based on this diffusion tendency.

  In this way, when a black image frame is detected from among a large number of frames constituting a moving image, the black image timing detection unit 212 generates the black image at any timing in the moving image. To detect. The timing expression method may be expressed by the number of frames from the initial frame or may be expressed by time intervals. In the following, the black image timing detection unit 212 expresses the black image generation timing by the number of frames. The generated timing signal is generated.

  As shown in FIG. 7, the character recognizing unit 213 searches the dictionary database 6A for timing information 501 having black image generation timing at the same timing as or similar to the timing signal 504 thus generated, and the searched timing information Is read from the database 6A. That is, the character recognition unit 213 determines whether the timing of the black image sequence 502 in the timing information 501 and the generation timing of the black image sequence 505 in the timing signal 504 are the same or similar. Is done. At this time, it is preferable that the black image 511 suddenly generated in the timing signal 504 is determined not to be based on the movement of the finger F but to be discarded (thinned out) based on the generation timing or the like. For example, when the black image 511 has an interval of 2 frames or less (1/15 seconds or less) with respect to the adjacent black image 505, the black image 511 can be determined as noise and thinned. Further, instead of simply discarding such a black image 511, the probability of determining that it is not a black image may be lowered.

Although the embodiments of the invention have been described above, the present invention is not limited to these embodiments, and various modifications and additions can be made without departing from the spirit of the invention. For example, in the above embodiment, a moving image is picked up by one camera 33 having one lens, but a plurality of time-synchronous or asynchronous moving images obtained by a camera having a plurality of lenses are selected. Alternatively, a plurality of series of images may be superimposed and used as a moving image. Furthermore, when superimposing images, they may be superimposed after multiplying by a separately determined weighting coefficient.
In the above-described embodiment, characters are input by hitting the lens of the camera 33, but it is needless to say that a control code such as a mail transmission command may be input.

When the character input mode according to the present embodiment is terminated, the key 35A is operated in the above example. Instead, for example, the lens of the camera 33 has a predetermined period of TE seconds (for example, about 2 to 5 seconds). ) When the character is continuously pressed, the character input mode of this embodiment may be terminated.
Further, as shown in FIG. 8, in the flip-type portable information terminal device in which the flip 34 and the main body 37 can be opened and closed by the hinge portion 38, the surface of the flip 34 (the surface seen when the flip is closed). In addition, a button 35B for starting or ending the character input mode of the present embodiment may be provided. Thereby, the character input mode of the present embodiment can be started or ended without opening the flip.

In the above embodiment, the example in which the lens of the camera 33 is hit with a finger F at a predetermined timing has been described. However, the lens may be hit with an object other than the finger F at a predetermined timing.
In addition, the pixel size may be various sizes or formats in the present invention. For example, ITU-T H.261 CIF, MPEG-1 SIF, NTSC image, PAL image, HDTV image, Any image of a different size may be used.

It is a block diagram which shows the circuit structure of the mobile telephone terminal concerning embodiment of this invention. It is a conceptual diagram which shows a mode that the character input mode by this Embodiment is performed. It is an example of the correspondence table | surface which shows the correspondence of the input character information stored in the dictionary database 6A, and timing information. An example of the operation of the motion vector calculation processing unit 2111 will be described. An example of a motion vector extraction region set in a frame is shown. An example of a motion vector extraction region set in a frame is shown. An example of operation | movement of the black image determination part 211 is shown. An example of the operation of the character recognition unit 2113 is shown. One of the modifications of this Embodiment is shown.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Radio part, 2 ... Baseband part, 3 ... Input / output part, 4 ... Power supply part, 5 ... Vibrator, 6 ... Memory card, 11 ... Antenna, 12 ... Antenna duplexer (DUP), 13 ... Reception circuit (RX), 14 ... Frequency synthesizer (SYN), 15 ... Transmission circuit, 21 ... Main control unit 21, 22 ... Demultiplexing unit, 23 ... Audio codec, 24 ... Image processing unit, 25 ... LCD control unit, 26 ... Memory unit, 31 ... Microphone, 32 ... Speaker, 33 ... Camera 34 ... LCD 35 ... Keyboard 41 ... Battery 42 ... Charging circuit 43 ... Voltage generation circuit 211 ... Black image determination unit 212 ... Black image Timing detector 213 ... Character recognition unit, 214 ... Timer, 2111 ... Motion vector calculation processing unit, 2112 ... Motion vector narrowing calculation processing unit, 2113 ... Black pixel content calculation processing unit.

Claims (8)

In a mobile terminal equipped with a function for inputting input information,
An imaging unit that captures a moving image;
A black image determination unit that determines whether an image included in the moving image is a black image;
A black image timing detection unit for detecting a generation timing at which the black image is generated;
A dictionary database that stores the occurrence timing and the input information in association with each other;
A reading unit that reads out the input information corresponding to the generation timing detected by the black image timing detection unit from the dictionary database ;
The black image determination unit includes a motion vector calculation unit that calculates a motion vector indicating a motion of a black pixel in a predetermined region on the image,
When the motion indicated by the motion vector is equal to or less than a predetermined amount, an image including the motion vector is determined as the black image.
A portable terminal characterized by that. The black image determining unit, the portable terminal according to claim 1, further comprising a calculation unit narrowing the motion vector to output a reference motion vector a plurality of motion vectors calculated by the motion vector calculating portion narrow in accordance with the criteria refine a given . The mobile terminal according to claim 2, wherein the motion vector narrowing calculation unit executes a process of removing disturbance data by absorbing variations in the motion vector calculated by the motion vector calculation unit. The mobile terminal according to claim 2, wherein the motion vector narrowing calculation unit removes disturbance data by performing thinning out of the images. The mobile terminal according to claim 2, wherein the motion vector narrowing calculation unit performs an edge detection of an object image in the motion image and executes a process of removing motion vectors other than the motion vector near the edge. When it is detected that a motion vector near the edge has diffused to the periphery of the image, the black image determination unit determines an image sequence of several frames with a small amount of motion vector immediately after that as the black image. The mobile terminal according to claim 5 . The black image determination unit further includes a black pixel content rate calculation unit that calculates the content rate of the black pixels in the image included in the moving image,
The portable terminal according to claim 1, wherein the image is determined as the black image when the motion indicated by the motion vector is equal to or less than a predetermined amount and when the content rate is equal to or greater than a predetermined value.   When the black image determination unit determines that the black image continues for a predetermined time or longer in the moving image, the input of the input information by the black image timing detection unit and the reading unit is started or ended. The mobile terminal according to claim 1, which is configured as follows.