JP4293358B2 - Mobile phone terminal with image communication function - Google Patents

Description

  The present invention relates to a mobile phone terminal with an image communication function suitable for a so-called videophone system, for example.

  Conventionally, various techniques as described below are known as techniques for displaying, for example, a person image taken by a television camera, for example, in the center of a television monitor.

  For example, Japanese Patent Application Laid-Open No. 2000-78564 discloses a labeling process in which a difference between frames of an image captured by a television camera is obtained and divided for each image area based on the difference image. A TV camera monitor person tracking method is disclosed in which a person's image is determined from the image area after the labeling process, an image of a certain range centered on the person image is cut out and displayed in the center of the TV monitor image. Has been.

  Further, for example, in Japanese Patent Application Laid-Open No. 9-46710 (Patent Document 2), when video data of a desired region of interest is extracted from video data captured by a camera unit and transmitted, the attention is paid. An image communication system capable of always obtaining a high-resolution image of the attention area by detecting the movement of the area and controlling the camera unit to zoom or change the shooting direction according to the movement of the attention area. An image communication apparatus is disclosed.

  Further, for example, in Japanese Patent Application Laid-Open No. 5-64182 (Patent Document 3), an area including a target object is detected from a video signal output from an object monitoring camera that captures a preset object monitoring area. There is disclosed a videophone imaging apparatus that detects a target and transmits a desired transmission video signal by controlling the direction and zoom ratio of a camera that captures an object based on the detection result.

JP 2000-78564 A (FIG. 1) Japanese Patent Laid-Open No. 9-46710 (FIG. 1) Japanese Patent Laid-Open No. 5-64182 (FIG. 1)

  By the way, in recent years, mobile phone terminals with an image communication function having a function of capturing a moving image and transmitting the captured moving image using a wireless telephone line are on the market. According to this mobile phone terminal with an image communication function, it is possible to send a moving image taken at various places such as outdoors to a communication partner terminal. In addition, in the mobile phone terminal with the image communication function, for example, in the case where the mobile phone terminal has a function that allows a monitor screen to be viewed with the camera lens facing the user's own direction, a so-called videophone system is provided. As described above, it is possible to make a call while looking at the appearance (for example, the face) of the other party on the monitor screen.

However, since a mobile phone terminal is generally used while being held in a hand, a photographed image tends to be blurred due to so-called camera shake. When there is a camera shake in this way, the image displayed on the monitor screen of the communication partner is also a blurred image, which is very difficult to see. In particular, when the mobile phone terminal with image communication function is used as a videophone, the user makes a call with the other party while photographing his / her own figure (for example, his face) with a camera. At this time, for example, if the user's own face is out of the shooting range of the camera due to camera shake or the like, the face on this side will not appear on the monitor screen of the other party's terminal, so the user will always be himself. It is necessary to make a call with the other party while taking a picture while paying close attention to the camera shake so that the face of the camera falls within, for example, the center of the shooting range of the camera. Also, when making a videophone call, the user must perform a special operation for the videophone call.

The present invention has been proposed in view of such circumstances, and can transmit a moving image without blurring to a communication partner, particularly when used as a videophone. A mobile phone with an image communication function that makes it possible to always display the appearance in the center of the screen of the other party's monitor, for example, and to prevent the user from performing special operations during videophone calls. The purpose is to provide a terminal.

Image communication function-equipped mobile phone terminal of the present invention, the photographing direction is changed by changing the direction of Rutotomoni objective lens with a large number of effective pixels of the imaging device than the transmission number of pixels in sending video data by the data communication transmitting an imaging means which is, a monitor means for projecting a moving image pickup means is taken on the screen, a photographing direction detection means for detecting a photographing direction of the imaging means, the moving image data taking means are taken The direction and amount of movement of the desired subject image within the range of the total number of effective pixels of the image sensor is detected every predetermined unit time, and the total number of effective pixels of the image sensor for each predetermined unit time. image processing means for cutting out an image of the number of transmission pixel at a position according to the direction and amount of the movement from the minute image, when photographing direction photographing direction detection means detects that the direction of the screen side of the monitor means Is set to enable the image cut-out function for the number of transmission pixels according to the direction and amount of movement in the image processing means, and when the shooting direction detected by the shooting direction detection means is a direction that is not on the screen side of the monitor means, A setting unit that disables an image clipping function for the number of transmission pixels according to the direction and amount of movement in the image processing unit, and the image clipping function for the number of transmission pixels is enabled The above-described problem is solved by transmitting, by data communication, moving image data composed of images corresponding to the number of transmission pixels cut out by the image processing means every predetermined unit time.

That is, the cellular phone terminal with an image communication function of the present invention uses the fact that the number of transmission pixels transmitted by data communication is smaller than the total number of effective pixels of the image sensor, and the subject image is all effective pixels due to, for example, camera shake. When moving within the range of the number, automatic image tracking of the subject image is performed by shifting the cutout position of the image for the number of transmission pixels within the range of the total number of effective pixels so as to match the direction and amount of the shaking movement. Realized. Further, in the present invention, when the shooting direction is the screen side direction of the monitor means, the image clipping function for the number of transmission pixels is set to be effective, while when the shooting direction is not the screen side of the monitor means, transmission is performed. The image cropping function for the number of pixels is disabled.

In the present invention, when the image for the number of transmission pixels is cut out from the image for the total number of effective pixels of the image sensor according to the direction and amount of movement of the subject image within the range of the total number of effective pixels of the image sensor. By changing the cut-out position, it is possible to transmit a moving image without blurring to the communication partner terminal. This makes it possible to always display the user's own figure, for example, in the center of the monitor screen of the other party terminal when used as a videophone. In the present invention, when the shooting direction is the screen side direction of the monitor means, the image cut-out function for the number of transmission pixels is set to be valid, while when the shooting direction is not the screen side of the monitor means, transmission is performed. By disabling the image cropping function for the number of pixels, the user does not have to perform a special operation during a videophone call.

  A mobile phone terminal with a videophone function, which is an embodiment of the mobile phone terminal with an image communication function of the present invention, will be described below with reference to the drawings.

[First embodiment of the present invention]
FIG. 1 is a schematic block diagram of a mobile phone terminal with a videophone function according to the first embodiment of the present invention.

  In FIG. 1, the audio signal input from the microphone of the microphone speaker unit 13 is band-limited by the audio processing unit 12, converted into digital audio data, and sent to the modem unit 11. The modem unit 11 performs a calling process according to a command from the CPU 17, and after establishing a communication line, converts the baseband digital voice data into an analog transmission signal and sends the analog transmission signal to the radio unit 10. The wireless unit 10 converts the analog transmission signal supplied from the modem unit 11 into a high-frequency signal in the radio frequency band and transmits the signal from the antenna.

  On the other hand, the received signal received by the antenna of the radio unit 10 is sent to the modem unit 11. The modem unit 11 converts a radio frequency reception signal in a radio frequency band into baseband digital audio data, and sends the digital audio data to the audio processing unit 12. The voice processing unit converts the digital voice data from the modem 11 into an analog voice signal and sends it to the speaker of the microphone speaker unit 13. Thereby, sound is output from the speaker.

  Further, in the mobile phone terminal of the present embodiment, for example, when a moving image is captured by the camera unit 14, the moving image signal input from the camera unit 14 is sent to the image processor 15. The image processor 15 performs predetermined digital processing on the moving image signal, and sends moving image data generated by the digital processing to the monitor unit 16. As a result, a moving image captured by the camera unit 14 is displayed on the screen of the monitor unit 16.

  Furthermore, the mobile phone terminal of the present embodiment can also transmit moving image data captured by the camera unit 14. However, when a moving image captured by the camera unit 14 is transmitted in real time using a wireless telephone line, the number of pixels of the moving image that can be transmitted is, for example, the capacity of the wireless telephone line or the standard of the communication channel. Due to the limitation, the total number of effective pixels of the image sensor of the camera unit 14 is reduced. Note that in the case of a videophone system using a mobile phone terminal that is already in practical use, the total number of effective pixels of the image pickup element of the camera unit is 640 × 480 dots of VGA (Video Graphics Array) size. The number of pixels of a moving image to be transmitted / received is 176 × 144 dots of QCIF (Quarter Common Intermediate Format) size. In this way, the number of moving image pixels that can be transmitted and received is less than the total number of effective pixels of the image sensor of the camera unit. This is because the number of moving image pixels that can be transmitted and received cannot be increased due to the restriction on the data communication capacity and the required frame rate.

  As described above, when the captured moving image data is transmitted in real time using the wireless telephone line, the CPU 17 sends an instruction to the image processor 15 to transmit the moving image in real time. At this time, the image processor 15 converts the number of pixels of the moving image taken by the camera unit 14 into the number of pixels that matches the restrictions such as the capacity of the communication path and the standard of the communication path, and the converted moving image. The image data is sent to the modem 11. The modem 11 generates an analog transmission signal from the baseband moving image data and sends the analog transmission signal to the wireless unit 10. Then, the wireless unit 10 converts the analog transmission signal supplied from the modem unit 11 into a high-frequency signal in the radio frequency band and transmits it from the antenna.

  On the other hand, the high-frequency received signal of the moving image that is transmitted in real time from the communication partner terminal and received by the antenna of the wireless unit 10 is sent to the modem unit 11 where it is demodulated into baseband moving image data, Sent to the processor 15. At this time, the image processor 15 sends the moving image data from the modem unit 11 to the monitor unit 16. As a result, the moving image transmitted from the call destination terminal in real time is displayed on the monitor unit 16.

  In the mobile phone terminal according to the present embodiment, the camera unit 14 includes a mechanism that can change the direction of the objective lens (that is, the shooting direction) to a desired direction. There are various mechanisms that can change the orientation of the objective lens in a desired direction as described above, and any of these various mechanisms can be applied to the mobile phone terminal according to the present embodiment. Therefore, the detailed explanation of the mechanism is omitted here.

  For example, in the mobile phone terminal of the present embodiment, when a moving image of a desired subject is shot, for example, as shown in FIG. 2, the orientation of the objective lens 21 of the camera unit 14 is directed toward the subject. Become. That is, when shooting a moving image of a subject in a state as in the example of FIG. 2, the user 30 views the subject image displayed on the screen of the monitor unit 16 while the subject image is displayed on the monitor unit 16. Shooting is performed while matching the position and orientation of the mobile phone terminal so that the subject comes to a predetermined position on the screen, that is, the subject enters a desired position within the shooting range of the camera unit 14.

  On the other hand, in the mobile phone terminal of the present embodiment, for example, when making a videophone call, the orientation of the objective lens 21 of the camera unit 14 is provided with the monitor unit 16 as shown in FIG. The user 30 is photographed by the camera unit 14 that is directed to the side.

  In this way, when a videophone call is performed, a moving image 31 sent from the call partner terminal is displayed on the screen of the monitor unit 16 as shown in FIG. The own moving image, that is, the moving image 42 taken by the camera unit 14 directed toward the user 30 as shown in FIG. 3 is displayed in the sub-window 41, for example. The sub window 41 is displayed in a small size and arranged at a predetermined position (or a desired position) so as not to interfere with the display of the moving image 31 of the other party on the screen of the monitor unit 16. It is.

  Therefore, when making a videophone call using the mobile phone terminal of the present embodiment, the user 30 can place the mobile phone terminal 20 at a position where the screen of the monitor unit 16 can be seen as shown in FIG. As shown in FIG. 4, while viewing the moving image 31 of the other party on the screen of the monitor unit 16 as shown in FIG. The mobile phone terminal is matched with the position and direction of the mobile phone terminal so that the mobile phone terminal is positioned approximately in the center of the shooting range of the camera unit 14 so that the mobile phone terminal 42 is reflected.

  Here, when moving image data captured by a mobile phone terminal is transmitted to a partner terminal, as described above, for example, a subject image (television television) blurred on the monitor screen of the partner terminal of the communication partner due to camera shake or the like. In the case of a telephone, the user 30 himself / herself) may be displayed, or the subject image may be displayed in a state of being off, for example, near the center of the monitor screen of the communication partner terminal. For this reason, in the mobile phone terminal of the present embodiment, the moving image captured by the camera unit 14 is transmitted in real time using the wireless telephone line as in the examples of FIGS. 2 and 3 described above. In such a case, a subject image (the user 30 in the case of a videophone) in a blurred state is displayed on the monitor screen of the communication partner terminal by the subject automatic position tracking function described below, For example, the image is prevented from being displayed in a state where the image is out of the vicinity of the center of the monitor screen of the communication partner terminal.

[Description of automatic subject tracking function]
Hereinafter, the subject automatic position tracking function when moving image data is transmitted in real time using a wireless telephone line will be described by taking the telephone call shown in FIGS. 3 and 4 as an example.

  In the present embodiment, when a moving image captured by the camera unit 14 is transmitted in real time via a wireless telephone line, the number of pixels of the moving image that can be transmitted depends on the capacity of the wireless telephone line and the standard of the communication path as described above. For example, the total number of effective pixels of the image sensor of the camera unit 14 is reduced. In the following description, among all the effective pixel areas of the image sensor of the camera unit 14, image areas for the number of pixels that include the subject image formed on the image sensor by the objective lens 21 and can be transmitted in real time, This is called a transmission image area.

  When the mobile phone terminal of the present embodiment transmits a moving image captured by the camera unit 14 in real time through a wireless telephone line, for example, as shown in FIG. From the inside, only the portion of the pixel number area 41 including the subject image whose image formation position is shaken due to camera shake is automatically tracked and cut out, and the cut-out image data is transmitted to monitor the communication partner terminal. A blurred subject image (user 30 in the case of a videophone) is displayed on the screen, or the subject image is displayed in a state of being off the center of the monitor screen of the communication partner terminal, for example. To prevent that.

  That is, as shown in FIG. 5, when the total number of effective pixels of the image sensor of the camera unit 14 is, for example, 64 × 64 dots, and the number of pixels in the transmission image area to be transmitted in real time is, for example, 32 × 32 dots, for example, I If the subject image (user's own image) 42a in the frame is at the position indicated by the solid line in the figure, and the subject image (user's own image) 42b in the frame I + 1 is at the position indicated by the dotted line in the figure, I It can be seen that the subject image has moved by two dots in the lower right direction of the figure between the frame and the I + 1 frame.

  Accordingly, after transmitting the data of the subject image 42a of the I frame, when transmitting the data of the subject image 42b of the I + 1 frame, the inter-frame difference between the I frame and the I + 1 frame is detected, and the interval between the frames is detected. Based on the difference, if the transmission image area 41b transmitted during the I + 1 frame is made an area moved by 2 dots in the lower right direction in the figure with respect to the transmission image area 41a transmitted in the previous I frame. The position of the subject image 42b in the transmission image area 41b at the time of the I + 1 frame is substantially the same as the position of the subject image 42a in the transmission image area 41a at the time of the I frame (for example, the center position of the transmission image area). ) Can be fixed. That is, in the example of FIG. 5, the inter-frame difference between the I frame and the I + 1 frame indicates that the subject image is shifted by, for example, the lower right 2 dots in the figure due to camera shake or the like. After transmitting the data of the transmission image area 41a in the I frame, when cutting out the transmission image area 41b including the subject image 42b of the I + 1 frame from all the effective pixel areas of the imaging device of the camera unit 14, If the shift is equivalent to 2 dots, the position of the subject image 42b in the transmission image area 41b of the I + 1 frame is fixed to be approximately the same as the position of the subject image 42a in the transmission image area 41a of the I frame. be able to.

  As a result, on the monitor screen of the counterpart terminal that has received the image data of the transmission image area 41a of the I frame and the image data of the transmission image area 41b of the next I + 1 frame, both the I frame and the I + 1 frame are subject images. The sub-window 41 in which the (source user's own image) 42 is substantially in the same position is displayed.

  Note that in the mobile phone terminal of the present embodiment, the image processor 15 performs processing such as detection of inter-frame differences and extraction of a transmission image area in accordance with inter-frame differences from all effective pixel areas of the image sensor of the camera unit 14. It is realized by the data processing inside. In addition, the image processor 15 recognizes the subject image by, for example, edge detection.

  As described above, according to the mobile phone terminal with the videophone function of the present embodiment, the moving image pixels that can be transmitted when the moving image captured by the camera unit 14 is transmitted in real time using the wireless telephone line. Using the fact that the number is smaller than the total number of effective pixels of the image sensor of the camera unit 14, the transmission image area cut out from the total number of effective pixels of the camera unit 14 is shifted based on the interframe difference. Thus, for example, the subject image can be automatically tracked only by data processing in the image processor 15 without moving the lens system, the camera unit 14 or the like.

  In the above description, real-time transmission of moving image data at the time of a videophone call shown in FIGS. 3 and 4 is given as an example, but the present invention is at the time of subject shooting as in the example of FIG. Needless to say, the present invention is also applicable to real-time transmission of moving image data.

[Second Embodiment of the Present Invention]
In the above-described embodiment, the example in which the subject automatic position tracking function is operated when moving image data is transmitted in real time in any of the cases of FIGS. 2 and 3 has been described. For example, as illustrated in FIG. When the direction of the 14 objective lens 21 is directed to the opposite side of the monitor unit 16, the subject automatic position tracking function is disabled. On the other hand, during the telephone call shown in FIG. The automatic subject tracking function may be automatically enabled only when moving image data captured in a state where the direction is directed to the side where the monitor unit 16 is provided is transmitted in real time. That is, if the object automatic position tracking function is automatically enabled when the objective lens 21 of the camera unit 14 is directed to the monitor unit 16 side, the user does not perform any special operation. During a videophone call, a blurred subject image (image of the user 30) is displayed on the monitor screen of the communication partner terminal due to camera shake or the like, for example, on the monitor screen of the communication partner terminal It is possible to avoid displaying the subject image (image of the user 30) out of the vicinity of the center.

  FIG. 6 is a schematic block diagram of the mobile phone terminal according to the second embodiment of the present invention, which can automatically switch the validity / invalidity of the subject automatic position tracking function depending on the direction of the objective lens 21 of the camera unit 14. The figure is shown. In FIG. 6, the same constituent elements as those in FIG. 1 are denoted by the same reference numerals as those in FIG. 1, and description thereof is omitted.

  In FIG. 6, the shooting direction detection unit 19 detects whether or not the direction of the objective lens 21 of the camera unit 14, that is, whether or not the direction of shooting by the camera unit 14 is the monitor unit 16 side. The detection signal of the photographing direction detector 19 is sent to the automatic position tracking function valid / invalid setting unit 50.

  When the automatic position tracking function valid / invalid setting unit 50 detects that the direction of the objective lens 21 of the camera unit 14 is the monitor unit 16 side based on the detection signal from the imaging direction detection unit 19, A setting for disabling the automatic position tracking function of the image processor 15 when outputting a control signal for enabling the automatic position tracking function and detecting that the direction of the objective lens 21 of the camera unit 14 is not on the monitor unit 16 side. Output a control signal.

  Accordingly, the cellular phone terminal having the configuration of FIG. 6 can automatically set the subject automatic position tracking function to be effective only when the objective lens 21 of the camera unit 14 is directed to the monitor unit 16 side.

[Third embodiment of the present invention]
In each of the embodiments of the present invention described above, only a configuration for performing automatic position tracking of a subject image by cutting out a transmission image area corresponding to a difference between frames from all effective pixel areas of the image sensor of the camera unit 14 is described. However, in addition to this, automatic position tracking of the subject may be performed by mechanically moving the image sensor of the camera unit 14 or the camera unit 14 itself.

  FIG. 7 shows a third embodiment of the present invention in which the transmission image area is cut out according to the inter-frame difference and the image sensor of the camera unit 14 or the camera unit 14 itself is also moved. 1 is a schematic block diagram of a mobile phone terminal of FIG. The configuration shown in FIG. 7 can automatically enable / disable the subject automatic position tracking function according to the direction of the objective lens 21 of the camera unit 14 as in the second embodiment. Yes. In FIG. 7, the same constituent elements as those in FIGS. 1 and 6 are denoted by the same reference numerals as those in FIGS. 1 and 6, and description thereof is omitted.

  In FIG. 7, the XY position moving device 51 mechanically moves the image sensor of the camera unit 14 or the camera unit 14 itself in the X-axis direction and the Y-axis direction based on the inter-frame difference detected by the image processor 15. It is made a device that can.

  In addition, the mobile phone terminal of the third embodiment uses the movement of the image sensor of the camera unit 14 or the camera unit 14 itself by the XY position moving device 51 for rough adjustment of the subject position automatic tracking. The transmission image area shift as in the first embodiment is used for fine adjustment of subject position automatic tracking.

  As described above, the mobile phone terminal having the configuration of FIG. 7 performs rough adjustment in which the image sensor of the camera unit 14 or the camera unit 14 itself is roughly moved by the XY position moving device 51 during subject position automatic tracking. In addition, by performing fine adjustment by shifting the transmission image area as described above, the position of the subject can be automatically and quickly tracked.

[Summary]
As described above, according to the mobile phone terminal of each embodiment of the present invention, when moving images captured by the camera unit 14 are transmitted in real time using a wireless telephone line, automatic position tracking of subject images is performed. By doing this, you can send a moving image without blurring to the destination terminal of the subject image. Especially when using it as a videophone, monitor the user's own appearance on the monitor of the destination terminal. It is possible to always display in the center of the screen.

  The above description of the embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and it is needless to say that various modifications can be made according to the design or the like as long as the technical idea according to the present invention is not deviated.

  For example, in the present invention, not only a mobile phone terminal but also a PDA (Personal Digital Assistant) having a camera function and a wireless communication function transmits moving image data captured by the camera function using a wireless telephone circuit or the like. It is also applicable to such cases.

It is a block diagram which shows schematic structure of the mobile telephone terminal with a video telephone function of the 1st Embodiment of this invention. FIG. 4 is a diagram used for explaining a shooting direction of a camera unit when shooting a moving image of a desired subject in the mobile phone terminal of the present embodiment. It is a figure used for description of the imaging direction of a camera part in the case of performing a telephone call by a video phone in the mobile phone terminal of the present embodiment. It is a figure which shows the example of a screen display of the monitor part at the time of the telephone call by the videophone in the mobile telephone terminal of this embodiment. It is a figure used for description of the subject automatic position tracking function according to the interframe difference in the mobile phone terminal of the present embodiment. It is a block diagram which shows schematic structure of the mobile telephone terminal with a video telephone function of the 2nd Embodiment of this invention. It is a block diagram which shows schematic structure of the mobile telephone terminal with a video telephone function of the 3rd Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Radio | wireless part, 11 Modem, 12 Sound processing part, 13 Microphone speaker part, 14 Camera part, 15 Image processor, 16 Monitor part, 17 CPU, 19 Shooting direction detection part, 20 Mobile phone terminal, 21 Objective lens, 30 User , 31 Calling party moving image, 40 All effective pixel area, 41 Sub window, 41a I frame transmission image area, 41b I + 1 frame transmission image area, 42 User moving image, 42a I frame subject image, 42b I + 1 Frame subject image, 50 automatic position tracking function enable / disable setting unit, 51 XY position moving device

Claims (2)

Rutotomoni with more effective number of pixels of the image sensor than the transmission number of pixels in sending video data by the data communication from the radio unit, a photographing means for photographing direction is changed by changing the orientation of the objective lens,
Monitor means for displaying on the screen a moving image taken by the imaging means;
Photographing direction detecting means for detecting the photographing direction of the imaging means;
When moving image data captured by the imaging unit is transmitted from the wireless unit by data communication, a desired subject image formed on the image sensor is within the range of the total number of effective pixels of the image sensor. The direction and amount of movement of the image sensor are detected every predetermined unit time, and from the image corresponding to the total number of effective pixels of the image sensor, the direction and amount of movement are adjusted for each predetermined unit time. Image processing means for cutting out the image for the number of transmission pixels at the position ;
When the shooting direction detected by the shooting direction detection means is the screen side direction of the monitor means, the image processing function for the number of transmission pixels corresponding to the moving direction and amount of the image processing means is enabled. When the shooting direction detected by the shooting direction detection unit is a direction that is not on the screen side of the monitor unit, a function of cutting out images corresponding to the number of transmission pixels according to the moving direction and amount of the image processing unit And setting means for setting to invalid,
A moving image composed of images corresponding to the number of transmission pixels cut out by the image processing unit every predetermined unit time when the image clipping function for the number of transmission pixels is set to be effective by the setting means. A mobile phone terminal with an image communication function for transmitting data from the wireless unit by data communication. The predetermined unit time is a frame time of a moving image,
2. The image communication function according to claim 1 , wherein the image processing unit obtains an inter-frame difference of images corresponding to the total number of effective pixels of the image sensor and detects the direction and amount of the movement based on the inter-frame difference. Mobile phone terminal.

Images