JP2008294815A - Information terminal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information terminal by which the displacement of a face image within a photographic range is grasped with sufficient convenience. <P>SOLUTION: A cellular phone 10 includes; an imaging part 12; a face detection part 14; a communication part 16; an image control part 18; and a display part 20, calculates a displacement amount of a picked-up face from a reference point on a screen of the display part 20 from an image picked up by the face detection part 14, outputs the calculated displacement amount as positional information, performs control by shifting a received image matched to the reference point received via the communication part 16 based on the positional information by the image control part 18 and outputs the controlled image on the display part 20. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an information terminal device, and the information terminal device of the present invention relates to, for example, a mobile phone or a PDA (Personal Digital Assistant) having an image display function.

  Conventionally, when a videophone device checks whether it is within the transmission screen of the other party's reception screen, for example, it displays the image on a part of the screen using the picture-in-picture method or the screen division method. A method and a method of switching and displaying a received image sent from the other party and a transmitted image obtained by capturing the subject are applied.

As another approach, as disclosed in the videophone device of Patent Document 1, the face area of the user is detected from the image data input from the camera, and the detected face area is located at the center of the screen. Systems have been devised that control the direction of the camera to be displayed and do not need to be adjusted by the user.
Patent 2947113 JP-A-8-251561 JP 2002-51316 A Patent 3593067

  However, when considering videophone communication by a small terminal device such as a mobile phone, the picture-in-picture method or the screen division method has a problem that it is difficult to confirm the display because the original screen is small. When switching between the partner's image and your own image, such a problem does not occur, but considering taking a picture with your hand like a mobile phone, even though the imaging position frequently shifts due to camera shake, In addition to confirming the display position only during switching, there is a problem that the operation is complicated.

  As an apparatus corresponding to these problems, there is a user interface apparatus for an image communication terminal disclosed in Patent Document 2. This device has a function of notifying the user, for example, using a change in the received image when the position of the user deviates from the shooting range. However, there is a problem that the user may be surprised by a sudden notification, and the deviation within the shooting range cannot be confirmed.

  In the case of Patent Document 1, such a problem does not occur. However, a mechanism for automatically controlling the direction of the camera on the mobile phone is technically difficult and expensive, which is realistic. It wasn't.

  It is an object of the present invention to provide an information terminal device that can eliminate such drawbacks of the prior art and can grasp a positional deviation of a face image within a shooting range with good convenience.

  In order to solve the above-described problems, the present invention provides an image capturing unit that captures an image of a user, a communication unit that transmits an image captured to a communication partner device, and receives an image from the communication partner device. An information terminal device including at least display means for displaying a received image and communicating the captured images with each other, wherein the information terminal device is a user from a reference point on the screen of the display means from the captured image. A face detection unit that calculates the amount of misalignment of the face at the face and outputs the calculated amount of misregistration as position information, and a control that shifts the received image that matches the reference point received via the communication unit based on the position information And an image control means for outputting the controlled image to the display means.

  In order to solve the above-described problems, the present invention provides an image capturing unit that captures an image of a user, a communication unit that transmits an image captured to a communication partner device, and receives an image from the communication partner device. An information terminal device including at least display means for displaying a received image and communicating the captured images with each other, wherein the information terminal device displays a screen of a display means for an image received from a communication partner device. Calculating the amount of deviation from the reference point in the face, and outputting the calculated position deviation amount as position information, and the deviation of the received image with respect to the reference point of the image received via the communication means based on this position information And image control means for outputting the controlled image to the display means.

  Further, in order to solve the above-described problem, the present invention transmits an image picked up to an image capturing unit for capturing an image of a user and an image of a communication partner device, and receives an image from the communication partner device. And an information terminal device that communicates the captured images with each other from the reference point on the screen of the display means. A face detection unit that calculates the amount of misalignment of the face of the user, outputs the calculated amount of misregistration as position information, and corrects and controls the amount of misregistration of the face in the image captured based on this position information. Image control means for outputting the captured image to the communication means, shifting the received image that matches the reference point received via the communication means based on the position information, and outputting the controlled image to the display means. It is characterized in.

  Furthermore, in order to solve the above-described problems, the present invention provides an imaging unit that images a user, an extraction unit that extracts facial feature data included in the captured image, a unit that stores character data, and a communication partner. A communication unit that transmits an image captured to a destination device and receives an image from a partner device to communicate with, an image creation unit that creates an image based on character data and facial feature data, and at least received Display means for displaying an image, and the extraction means sets a predetermined size for the face image in the captured image and the received image as each reference size, and receives the captured image for the reference point on the screen of the display means and the reception The amount of displacement of the face image in the captured image is calculated, and the ratio of the captured image and the received image to each reference size of the face image is calculated The calculated positional deviation amount and ratio of the face of the user and the positional deviation amount and ratio of the face in the received image are output as positional information to the image creating means, and the image creating means captures an image captured based on the positional information. Adjusting the ratio of the face displacement amount and the reference size with respect to the image, and adjusting the ratio of the function to output to the communication means and the face displacement amount and the reference size with respect to the received image based on the position information And a function of outputting to the display means, and the captured images are communicated with each other.

  According to the information terminal device of the present invention, the amount of misalignment of the face of the user from the reference point on the screen of the display unit is calculated from the image captured by the face detecting unit, and the calculated amount of misregistration is output as position information. Then, the image control means controls the received image that matches the reference point received via the communication means on the basis of this position information, and outputs the controlled image to the display means so that the position of the user's face Because the misalignment can be grasped by the display image displaying the face of the other party to communicate with, it is very easy to check and very convenient, and the alignment itself is reflected in the mirror Since they are equivalent, they can be provided as functions that are very intuitive, easy to understand, and easy to use.

  According to the information terminal device of the present invention, the deviation amount from the reference point on the screen of the display means of the image received from the counterpart device communicating with the face detection means is calculated, and the calculated positional deviation amount is used as the position information. The image control means controls the deviation of the image received via the communication means based on this position information, and outputs the controlled image to the display means. Since it is possible to display the image with the display deviation corrected, it is possible to provide a comfortable image communication because the face of the communicating party can be displayed without blurring.

  In addition, according to the information terminal device of the present invention, the amount of misalignment of the face of the user from the reference point on the screen of the display unit is calculated from the image captured by the face detecting unit, and the calculated amount of misregistration is used as the position information. The image control unit corrects and controls the amount of face misalignment in the image captured based on this position information by the image control unit, outputs the corrected captured image to the communication unit, and passes the communication unit based on the position information. By controlling the received image that matches the reference point to be received by shifting and outputting the controlled image to the display means, even if the image captured by the user is shaken frequently and the face position shifts frequently, the image to be transmitted Since the face displacement amount is calculated and corrected, the terminal device of the communication partner can receive an image with no face displacement. In particular, when communicating with a videophone using a terminal device that you hold and operate like a mobile phone, you can provide a communication-free image of the face position and communicate the user's face position shift. This makes it possible to grasp the face of the other party to be displayed based on the displayed image, so that good image communication can be realized.

  Furthermore, according to the information terminal device according to the present invention, the extraction unit sets a predetermined size for the captured image and the face image in the received image as each reference size, and images the reference point on the screen of the display unit. The amount of deviation of the position of the face image in the image and the received image is calculated, the ratio of the captured image and the size of the face image in the received image to each reference size is calculated, and the amount of deviation of the face position and the ratio in the calculated user And the position displacement amount and ratio of the face in the received image are output as position information to the image creation means, and the image creation means adjusts the ratio of the face displacement amount and the reference size with respect to the image captured based on the position information. Then, the feature data is output to the communication means, and the face displacement amount and the reference support for the feature data received based on the position information are output. By outputting the image data created by adjusting the ratio to the image to the display means, the processing of the face image and background image displayed by the selected character is separated, and position correction and display magnification are applied only to this face image. It can be corrected. For this reason, when shifting the position of the actual captured image, there may be a problem of display of a portion outside the imaging area that may occur due to movement of the image itself, or when the image is enlarged / reduced, the background is enlarged / reduced. Comfortable image communication can be realized without causing problems.

  Next, an embodiment of an information terminal device according to the present invention will be described in detail with reference to the accompanying drawings. Referring to FIG. 1, an embodiment of an information terminal device according to the present invention includes an imaging unit 12, a face detection unit 14, a communication unit 16, an image control unit 18, and a display unit 20, from an image captured by the face detection unit 14. The position displacement amount of the captured face from the reference point on the screen of the display unit 20 is calculated, the calculated position displacement amount is output as position information, and the image control unit 18 uses the position information as a basis via the communication unit 16 The received image that matches the reference point to be received is shifted and controlled, and the controlled image is output to the display unit 20, so that the misalignment of the user's face can be grasped from the display image on which the face of the other party to communicate is displayed. Because it is possible to confirm, it is very easy to check and very convenient, and the alignment itself is equivalent to mirroring your face, so it is very intuitive, easy to understand and easy to use Can be offered as.

  In this embodiment, the information terminal device of the present invention is applied to a mobile phone 10. The illustration and description of parts not directly related to the present invention are omitted. In the following description, the signal is indicated by the reference number of the connecting line in which it appears.

  As shown in FIG. 1, the mobile phone 10 includes an imaging unit 12, a face detection unit 14, a communication unit 16, an image control unit 18, and a display unit 20. The imaging unit 12 forms an image of incident light from an object field via an optical system on an imaging surface of a CCD (Charge Coupled Device), for example, and performs photoelectric conversion with a CCD light receiving element, for example, a CCD (Charge Coupled Device) It has a function of obtaining an image signal on the imaging surface. The imaging unit 12 digitizes the image signal generated by the photoelectric conversion, and outputs the image data 22 to the face detection unit 14 and the communication unit 16.

  The face detection unit 14 has a function of detecting a face position from the captured image data 22 and calculating positional deviation information. The face detection unit 14 extracts the outline of the face from the captured image data 22, calculates the center position of the face, and calculates the shift amount from the center of the imaging screen to the center position of the face as position shift information. The geometry of the extracted face outline will be described in detail later. The face detection unit 14 supplies the detected positional deviation information 24 to the image control unit 18.

  The communication unit 16 is connected to a communication network 26 such as a public telephone network, a mobile phone network, or the Internet network, for example, and has a function of transmitting / receiving voice data, image data, and input data to / from a communication destination. The communication unit 16 receives audio data, image data, and information that is input data supplied from the communication partner, and outputs the received image data 28 to the image control unit 18.

  In the present embodiment, voice is simultaneously transmitted and received by a voice input / output unit (not shown), but the disclosure is omitted because it is not directly related to the present invention.

  The image control unit 18 has a function of controlling the display position of the supplied image data. The image control unit 18 outputs the controlled display image data 30 to the display unit 20. The display unit 20 has a function of displaying the supplied display image data 30 using, for example, a liquid crystal display.

  Next, the operation of the mobile phone 10 to which the information terminal device of the present invention is applied will be described. The mobile phone 10 has a videophone function for transmitting and receiving face images to each other. The cellular phone 10 takes an image including the user's face with the imaging unit 12 and sends the image data 22 to the communication unit 16. An example of imaging is shown in FIG. It is assumed that the subject 32 as the user is in the angle of view 34 that is the imaging range of the imaging unit 12. In this embodiment, the subject 32 is shown on the left side as viewed from the imaging unit 12 side.

  The communication unit 16 transmits the image data 22 to the communication device with which it communicates. The face detection unit 14 detects the position of the face from the image data 22 supplied from the imaging unit 12, and detects the amount of deviation from the reference coordinates of the captured image. In this embodiment, the reference coordinates are set as the center coordinates.

  Various face detection methods can be used, such as those based on skin color extraction that uses color difference information and luminance difference information for face contours, those that use facial part extraction such as eyes and mouth, and those that use template matching. can do. Patent Document 1 discloses an example of face detection by skin color extraction.

Next, detection of the displacement amount of the captured face position will be described. FIG. 3 shows a case where the image data 22 captured in FIG. 2 is displayed on the display unit 20 as it is. In this embodiment, the horizontal direction on the screen of the display unit 20 is set as the x axis, and the vertical direction is set as the y axis. Also, set the horizontal width of the screen to x _d and the vertical width to y _d . In this embodiment, as shown in FIG. 3, the origin (0, 0) is set at the upper left corner. Therefore, the center 36 of the screen is (S _x , S _y ) = (x _d / 2, y _d / 2). The face detection unit 14 uses a face detection technique to detect a face area in the captured image data 22, and the boundary of the face area, left end x1, right end x2, upper end y1, lower end y2, left end x _L , right end x _R , The upper end y _U and the lower end y _D are calculated. From this value, the center 38 of the face area is obtained as (F _cx , F _cy ) = ((x1 + x2) / 2, (y1 + y2) / 2). Therefore, when the face area center 38 and the screen center 36 coincide with each other and the face position shift amount is zero, the face position shift amount 24 is ((x1 + x2) / 2- _xd / 2, (y1 + y2) / 2- y d / 2) = ((x1 + x2) / 2-S x, represented by (y1 + y2) / 2- S y). When this positional deviation amount 24 is displayed as a vector, the positional deviation amount in FIG. 3 can be displayed as a vector 40.

The image control unit 18 performs image control on the received image sent from the communication partner via the communication unit 16. In the present embodiment, in order to simplify the description, the face position of the communication partner included in the image from the communication partner supplied to the communication unit 16 is set to be at the center 36 of the screen. The received image in this case is shown in FIG. In this case, a ((x3 + x4) / 2 , (y3 + y4) / 2) = (x d / 2, y d / 2) = (S x, S y). The image control unit 18 performs shift control on the image data 28 received from the counterpart device by the amount of the face displacement 24 of the subject 32 calculated by the face detection unit 14, and displays the image on the display unit 20. .

Here, one point to note is that even when the screen is shifted to the right with respect to the center of the screen of the imaging unit 12 when viewed from the subject 32 side, the captured image is shifted to the left as shown in FIG. It will be reflected. Positional deviation amount 24 of the face calculated by the face detecting section 14, ((x1 + x2) / 2-x d / 2, (y1 + y2) / 2-y d / 2) = ((x1 + x2) / 2-S _x , (y1 + y2) / 2-S _y ). However, considering user convenience, if the user is shifted to the right like a mirror in which the user is shown, the display unit 20 is intuitively easy to understand if the image is shifted to the right. Therefore, the shift control amount for image received by the image control unit 18, (- (x1 + x2) / 2 + x d / 2, (y1 + y2) / 2-y d / 2) = (- (x1 + x2) / 2 + S _x , (y1 + y2) / 2−S _y ), and it is preferable to reverse the x-axis direction. As a result, the display unit 20 displays a display screen after the display position is corrected, as shown in FIG.

If the positional deviation amount 24 in this case is displayed as a vector, it can be displayed as a vector 42 indicating the positional deviation amount. The vector 40 in FIG. 3 and the vector 42 in FIG. 5 have a line-symmetric relationship with respect to x = x _d / 2. Therefore, when the subject 32 is shifted to the right side with respect to the imaging unit 12 as shown in FIG. 2, as shown in FIG.

  As described above, since the position shift of the subject 32 or the user's face is reflected on the face position of the received image sent from the communication partner, the user of the mobile phone 10 can display it. It is possible to grasp the positional deviation of the face from the face position of the image. In the picture-in-picture method and the screen division method, a small screen for displaying oneself is prepared. However, when considering the use with a mobile phone, there is a problem that the screen becomes too small to be confirmed. However, since the mobile phone 10 to which the present invention is applied can grasp its own position shift at the display position of the face image supplied from the communicating party, confirmation is very easy.

  In addition, compared with a case where the transmission image and the reception image are switched, it is possible to obtain the same confirmation effect without performing a complicated operation such as switching. Considering the case where the mobile phone 10 having the videophone function is used, since the camera is held in the hand, the imaging position is likely to be shifted, and it is desirable that the user can always check his / her imaging position. Since it is normal to make a call while activating the videophone function and looking at the face of the other party, the present invention that can grasp the imaging position of the person from the face position of the other party is very convenient. Furthermore, since the alignment itself is equivalent to mirroring one's face, it can be provided as a function that is very intuitive, easy to understand, and easy to use.

  Next, a modified example of the mobile phone 10 will be described. Although the present embodiment is the same component as the previous embodiment, the connection relationship is different. That is, as shown in FIG. 6, the mobile phone 10 of the present embodiment supplies the image data 28 received from the other party communicating with the communication unit 16 to the face detection unit 14 as well as the image control unit 18. Is different. As a result, this embodiment can also detect the face position of the image received from the communicating party.

Next, the operation of this embodiment in the mobile phone 10 will be briefly described. As shown in FIG. 7, the screen example of the mobile phone 10 according to the present embodiment shows a case where there is a positional displacement of the face supplied from the communicating party. In this case, the amount of face displacement 24 in the center 36 of the screen and the received image is the same as in the previous embodiment, and ((x7 + x8) / 2- _xd / 2, (y7 + y8) / 2- _{y d / 2) = ((} x7 + x8) / 2-S x, is calculated as (y7 + y8) / 2- S y). Thus, the cellular phone 10, the received image - ((x7 + x8) / 2-x d / 2, (y7 + y8) / 2-y d / 2) = - ((x7 + x8) / 2-S If it is shifted to _x , (y7 + y8) / 2−S _y ), the face position center 38 in the received image coincides with the screen center 36 in the display unit 20.

  Therefore, by operating in this way, as shown in FIG. 7, the reception screen when there is a positional deviation is equivalent to the reception screen when there is no positional deviation shown in FIG. Since other operations are the same as those in the previous embodiment, description thereof is omitted. In the previous embodiment, in order to simplify the description, the face position of the communication partner included in the image from the partner of communication supplied to the communication unit 16 is at the center 36 of the screen as shown in FIG. It was set as a thing. However, if the communicating party is also using a mobile phone, it is hand-held, so the face position of the communicating party is likely to fluctuate frequently. In the previous embodiment, the face position is shifted. I couldn't determine whether the other person's face was misaligned. In this embodiment, even if the face position of the communicating party is shifted by inputting the image data 28 supplied from the communicating party to the face detecting unit 14, the amount of face position shift can be calculated and corrected. . Based on this correction, if the same processing as in the previous embodiment is performed, the user must grasp the actual position of his face from the position of the other party's face on the display screen, not due to the face position of the other party. Is possible. In particular, the communication partner can effectively grasp the deviation of his / her face position when the videophone function is executed by a terminal device which is held and operated like a mobile phone with a camera function.

  Next, a modified example of the mobile phone 10 will be described. Although the present embodiment is the same component as the previous embodiment, the connection relationship is different. That is, as shown in FIG. 8, the mobile phone 10 of this embodiment not only supplies the image data 22 from the imaging unit 12 to the face detection unit 14, but also supplies the image data 22 to the image control unit 18. . The image control unit 18 is different in that the image data 44 is supplied to the communication unit 16 as data to be transmitted to a communicating party.

  Next, the operation of this embodiment in the mobile phone 10 will be briefly described. The imaging unit 12 sends the captured image to the face detection unit 14 and the image control unit 18. The face detection unit 14 calculates face position shift information 24 as a face position shift amount based on the image data 22 and outputs the face position shift information 24 to the image control unit 18.

  The image control unit 18 performs shift control on the image data 22 based on the supplied face positional deviation information 24, and sends image data 44 in which the face positional deviation is eliminated by the shift control to the communication unit 16. The image control unit 18 outputs the received image data 30 including the image of the communication partner to the display unit 20. As a result, the image to be transmitted is always an image with no face displacement, and comfortable image communication can be performed.

  As described above, in this embodiment, the image control unit 18 controls the image data 22 based on the positional deviation information 24, so that, for example, the user sits in front of a PC (Personal Computer) and has a built-in USB (Universal Serial Bus). When using a camera to make a videophone call, that is, image communication, it is considered that there is little problem with the change in the face position in the captured image, but when using the mobile phone 10, the position change of the face is frequently caused by holding. The face position of the captured image is frequently shifted in the configuration of the previous embodiment, and it is considered that the communicating party is very difficult to see the image.

  However, in the present embodiment, even if a face position shift frequently occurs due to camera shake of an image captured by the user, the transmitted image calculates and corrects the amount of face position shift. Then, it is possible to receive an image with no face displacement. In particular, when videophone communication is performed with a terminal device that is held and operated by a hand such as the mobile phone 10, it is possible to provide an image with no face position deviation to the communicating party and to exhibit good image communication. it can. In the example of FIG. 1, in order to grasp the deviation of the face position of the user, it is assumed that the face position of the image sent from the partner does not deviate, but this embodiment can be realized.

  Next, a modified example of the mobile phone 10 will be described. Although the present embodiment is the same component as the previous embodiment, the connection relationship is different. That is, in the mobile phone 10 of the present embodiment, as shown in FIG. 9, the communication unit 16 supplies the image data 28 received from the communicating party not only to the image control unit 18 but also to the face detection unit 14. Is different. This embodiment corrects the imaging position shift of the user in the image to be transmitted and transmits the image, and also corresponds to the case where there is a display shift of the communication partner in the received image.

Next, the operation of this embodiment in the mobile phone 10 will be briefly described. When the face area center 38 and the screen center 36 coincide with each other, the face detection unit 14 sets the face displacement amount to zero, and the face displacement amount ((x _L + x _R ) / 2−S _x , (y _U + y _D ) / 2−S _y ). As for the operation on the transmission side, the image control unit 18 captures an image captured so as to eliminate the positional deviation of the user in the image data 22-((x _L + x _R ) / 2-S _x , (y _U + y _D ) / 2−S _y ) = (− (x _L + x _R ) / 2 + S _x , − (y _U + y _D ) / 2 + S _y ) is shifted and sent to the communication unit 16 as an image to be transmitted. Thereby, in the transmission image, the positional deviation at the time of imaging is corrected, and the image displayed in the middle of the user is transmitted.

Next, the operation on the receiving side will be described. The image received by the communication unit 16 is sent to the face detection unit 14 and the image control unit 18. FIG. 10 shows a reception screen when the received image is supplied to the display unit 20 and displayed without using the image control unit 18. In this case, the amount of misalignment of the face of the communication partner is ((x _RL + x _RR ) / 2−S _x , (y _RU + y _RD ) / 2−S _y ). The subscript R indicates reception. The image control unit 18 performs shift control on the image received by the communication partner by the amount of the face displacement of the communication partner calculated by the face detection unit 14 so that the communication partner's face position is in the middle. Control the image 28. In this case, the shift control amount is ((x _RL + x _RR ) / 2−S _x , (y _RU + y _RD ) / 2−S _y ).

As in the previous embodiment, when the user is shifted to the right as in the mirror in which the figure is reflected, the image controller 18 is more intuitive because it is easier to understand that the image is shifted to the right on the display unit 20. The shift control amount with respect to the image received at is inverted in the x-axis direction and becomes (− (x _L + x _R ) / 2 + S _x , (y _U + y _D ) / 2−S _y ). The shift control amount to be finally added to the image received from the communication unit 16 is (− (x _L + x _R ) / 2 + S _x , (y _U + y _D ) / 2−S _y ) − ((x _RL + x _RR ) / 2-S _x , (y _RU + y _RD ) / 2-S _y ) = ((-x _L -x _R -x _RL -x _RR ) / 2 + 2S _x , (y _U + y _D -y _RU -y _RD ) / 2).

  As described above, according to the present embodiment, control is performed so that the face position deviation of the user is reflected in the face position of the image received from the communicating party without being affected by the face shift of the communicating party in the received image. Thus, the user can grasp the deviation of his / her face position from the face position of the other party to communicate in the display image.

  Furthermore, since the user corrects the user's face deviation in the captured image and sends the image to the device of the communicating party, the communicating party can see the user's face image in the center of the screen, Comfortable image communication can be performed.

  During video communication such as videophone, it is normal to talk while looking at the faces of the other parties communicating with each other, so it is very convenient in that you can grasp your imaging position from the face position of the communicating party Function can be provided. In addition, since the positioning itself can be handled in the same way as when the user's face is reflected in the mirror, it is possible to provide a very intuitive and easy-to-understand operation.

  Next, a modified example of the mobile phone 10 will be described. Although the present embodiment has the same components as the first embodiment, the image control section 18 includes a display magnification control function section 46 as shown in FIG. The display magnification control function unit 46 has a function of controlling the display magnification for an image to be displayed by supplying detection information 240 including face size information obtained based on the face position information detected by the face detection unit 14. .

  FIG. 12 shows a reference imaging screen when the user is imaged at an appropriate size, and FIG. 13 shows an imaging screen when the user is actually imaged. Further, in order to simplify the description in this embodiment, it is assumed that the received image is a reference reception screen example in which an image of a communication partner is shown in an appropriate size as shown in FIG.

First, as shown in FIG. 12, in the reference imaging screen, the width of the face size is (x _SSR -x _SSL ) and the vertical width is (y _SSD -y _SSU ). Here, the positions x _SSR , x _SSL , y _SSD and y _SSU indicate the right end, the left end, the lower end and the upper end of the captured face to be transmitted as a reference, and these values are calculated by the face detection unit 14.

Next, as shown in FIG. 13, the actual imaging screen is when the user places the imaging unit 12 closer to the reference position, and as a result, an image whose face is larger than the reference is obtained. . Here, the positions x _SR , x _SL , y _SD and y _SU indicate the right end, left end, lower end and upper end of the captured face to be transmitted, and these values are calculated by the face detection unit 14. At this time, the face size is calculated by the face detection unit 14 as a horizontal width (x _SR -x _SL ) and a vertical width (y _SD -y _SU ). Basically, (x _SR -x _SL ) / (x _SSR -x _SSL ) = (y _SD -y _SU ) / (y _SSD -y _SSU ), so in calculating the ratio with the reference captured image May use only information in one of the horizontal and vertical directions. In this embodiment, it is set to use information in the vertical direction. The face detection unit 14 is captured from the captured image in FIG. 13 at a magnification (y _SD −y _SU ) / (y _SSD −y _SSU ) compared to the face size in the reference in FIG. Calculated.

The display magnification control function unit 46 receives information 240 of the imaging magnification (y _SD −y _SU ) / (y _SSD −y _SSU ) from the face detection unit 14. Here, the positions x _SRR , x _SRL , y _SRD and y _SRU on the reference reception screen indicate the right edge, left edge, lower edge and upper edge of the face in the image received as the reference, and these values are calculated by the face detection unit 14. Is done. As shown in FIG. 14, the reference reception screen is calculated such that the width of the face size is (x _SRR -x _SRL ) and the vertical width is (y _SRD -y _SRU ). Further, positions x _RR , x _RL , y _RD and y _RU in the display image shown in FIG. 15 indicate the right end, left end, lower end and upper end of the face in the image to be displayed.

As shown in FIG. 15, the size of the face is displayed as (x _RR -x _RL ) = (x _SRR -x _SRL ) * (y _SD -y _SU ) / (y _SSD -y _SSU ), Enlarge the display so that (y _RD -y _RU ) = (y _SRD -y _SRU ) * (y _SD -y _SU ) / (y _SSD -y _SSU ). As a result, when the user gets too close to the imaging unit 12 and the size of the face on the imaging screen increases, the size of the face image from the other party on the display screen is also displayed.

  Since the face size in the image from the other party communicating on the display screen is displayed following the user's imaging distance, the appropriate distance depends on the face size in the image from the other party communicating on the display screen. It becomes possible to grasp whether or not the image is captured with the. During an image communication such as a videophone call, it is normal to talk while looking at the face image supplied by the other party. It is possible to provide highly convenient functions. In the case of a system with a small screen such as a videophone by the mobile phone 10, particularly effective effects can be exhibited.

  The mobile phone 10 disclosed next has the connection relationship of the mobile phone 10 shown in FIG. Further, as shown in FIG. 16, the image control unit 18 in the present embodiment includes the display magnification control function unit 46 described in the previous embodiment. An example of a received image sent from the communication partner is shown in FIG. In this example, the image of the other party communicating is smaller than the reference received image.

Next, the operation of this embodiment in the mobile phone 10 will be described. The face displacement correction is the same as in the previous embodiment. First, display magnification control on the transmission side will be described. Here, the captured image is set in the case of the image of FIG. In this case, the ratio of the face size in the captured image as the reference in FIG. 12 and the captured image in FIG. 13 is calculated as (y _SD −y _SU ) / (y _SSD −y _SSU ). When the image of FIG. 13 is sent as the transmission image, the face is too large to be seen on the other party's device for communication.

Therefore, in this embodiment, the display magnification control function unit 46 controls the reciprocal of (y _SD −y _SU ) / (y _SSD −y _SSU ) as the display magnification for the image in FIG. Thus, the vertical width of the face becomes (y _SD −y _SU ) * (1 / ((y _SD −y _SU ) / (y _SSD −y _SSU ))) = (y _SSD −y _SSU ). Therefore, the size of the face in the transmission image is transmitted with the same size as the captured image as the reference in FIG.

  Next, control of display magnification for an image supplied from the receiving side will be described. The received image in this case is shown in FIG. Since the received image is smaller than the size of the face in the reference image, it is corrected to the reference size.

The ratio of the face size between the received image in FIG. 17 and the reference received image in FIG. 14 is calculated as (y14−y13) / (y _SRD −y _SRU ). The display magnification control function unit 46 controls the reciprocal of the calculated ratio as the display magnification. The received image in FIG. 17 is (y14−y13) * (1 / ((y14−y13) / (y _SRD −y _SRU ))) = (y _SRD −y _SRU ). The received image in FIG. 17 has the display magnification controlled to the same size as the face size in the received image as a reference in FIG.

Furthermore, the mobile phone 10 controls the display magnification of the image captured by the user to the image controlled by the display magnification. The ratio of the captured image is calculated as (y _SD −y _SU ) / (y _SSD −y _SSU ) compared with the reference face size. Therefore, the display magnification control function unit 46 applies this value to the received image. In summary, the display magnification for the received image in FIG. 17 is ((y _SD -y _SU ) / (y _SSD -y _SSU )) * (y _SRD -y _SRU ) = (y _SD -y _SU ) * (y _SRD -y _SRU ) / (y _SSD -y _SSU ). As a result, the display image is displayed on the display unit 20 at the ratio of the captured image regardless of the size of the received image.

  Needless to say, the control of the display magnification is not limited to the previous embodiment or the present embodiment, and can be applied to an embodiment that is not described regarding the control of the display magnification.

  In the present embodiment, it is possible to reflect the position displacement of the face of the user in the face position of the communicating party, and further reflect the size of the face captured by the user in the size of the face of the communicating party. Therefore, it is possible to make a call while looking at the other party communicating through image communication such as a videophone, and to naturally move his / her face position forward / backward / left / right to an appropriate place. In particular, when performing image communication using a handheld camera such as a mobile phone, an effective effect can be exhibited.

  Finally, another embodiment of the cellular phone 10 to which the information terminal device of the present invention is applied will be described. In the embodiments so far, the display position of the captured image or the received image itself is shifted or displayed by the control for enlarging / reducing. In this embodiment, unlike these, the transmitting side extracts facial feature points from the captured image and sends the extracted feature data, and the receiving side uses a system for creating a display image based on the feature data and character data. set to target. This system is disclosed in, for example, “Information terminal device with image communication function and image distribution system” of Patent Document 4.

  As shown in FIG. 18, the mobile phone 10 in this embodiment includes an imaging unit 12, a communication unit 16, and a display unit 20, and includes a face feature extraction unit 48, a display image creation unit 50, and a character data storage unit 52. The face feature extraction unit 48 has a function of extracting facial feature points from the captured image. The face feature extraction unit 48 supplies transmission image data 54 representing facial feature points extracted from the captured image to the communication unit 16. Further, the face feature extraction unit 48 calculates information 240 regarding the face position and the face size based on the face feature points obtained from the captured image, and supplies the information 240 to the display image creation unit 50.

  The display image creation unit 50 has a function of creating a display image based on the feature data 28a and the character data 56 supplied from the receiving side. The display image creation unit 50 has a display position shift and enlargement / reduction control function based on the supplied information 240. The display image creation unit 50 outputs the created display image data 30 to the display unit 20. The character data storage unit 52 is a memory and stores character data. The character data storage unit 52 supplies the selected character data 56 to the display image creation unit 50.

  Next, the operation of the mobile phone 10 in this embodiment will be described. The face feature extraction unit 48 extracts, for example, eyebrows, eyes, mouth, nose, and contour from the supplied captured image data 22 as face feature information. The face feature extraction unit 48 transmits the position information or change information of the extracted data as transmission image data 54 via the communication unit 16 to the counterpart device with which it communicates.

  The display image creation unit 50 creates a display image based on the received image data 28a as the facial feature information sent from the communication unit 16 and the character data 56 from the character data storage unit 52. Further, the display image creation unit 50 corrects the position and display magnification of the face in the created display image based on the information 240 including the face position information and the face size information sent from the face feature extraction unit 48.

Assume that face feature data 28a of an image having no positional deviation shown in FIG. 4 is sent as data to be received from a communicating apparatus. Assuming that the user's face position is shown in FIG. 3, in this case, the face displacement amount is calculated as ((x1 + x2) / 2−S _x , (y1 + y2) / 2−S _y ). Here, for the convenience of the user, the left and right shifts are reversed and displayed. Therefore, the correction amount on the actual display screen is (− (x1 + x2) / 2 + S _x , (y1 + y2) / 2−S _y ). A display image in this case is shown in FIG.

  The display image creating unit 50 creates display data based on the character data 56 as shown in the figure. In this embodiment, the received face image is converted into a cat and displayed on the display unit 20.

  Here, the character data is data representing a character appearing in an animation, that is, a person, or data created by anthropomorphizing, for example, animals and objects. The display image creation unit 50 creates an image contributing to the conversation by reflecting the facial expression of the user who communicates using the character data 56.

In this embodiment, the display position is ((x20 + x21) / 2−S _x , (y20 + y21) / 2−S _y ) = (− (x1 + x2) / 2 + S _x , (y1 + y2) / 2−S _y ).

  In the present embodiment, in addition to reflecting the positional deviation of the face of the user in the face position of the other party to be communicated in the same manner as the previous embodiment, the size of the face captured by the user is further communicated. It is possible to reflect the size of the face. This makes it possible to talk while looking at the other party through image communication, and to move the position of his face naturally to an appropriate place. In particular, he makes a videophone call using a handheld camera such as the mobile phone 10. It is effective in the case.

  Furthermore, as in the previous embodiment, when shifting the position of the actual captured image, there is a problem in the display of the part outside the imaging area to move the image itself, and when enlarging / reducing, it is enlarged / reduced to the background. There is a problem of uncomfortable feeling due to reduction. However, in this embodiment, the processing of the face image and the background image displayed by the selected character can be separated, and the position correction and the display magnification can be corrected only for this face image. do not do.

It is a block diagram which shows the schematic structure of the mobile telephone to which the information terminal device which concerns on this invention is applied. It is a figure which shows an example of the imaging with respect to the to-be-photographed object by the imaging part of FIG. It is a figure which shows the screen of the captured image imaged with the imaging part of FIG. It is a figure which shows the screen of the received image without the position shift displayed on the display part of FIG. It is a figure which shows the display screen after position shift correction displayed on the display part of FIG. It is a block diagram which shows the schematic structure of the mobile telephone to which the information terminal device which concerns on this invention is applied. It is a figure which shows the screen of a received image when there exists a position shift displayed on the display part of FIG. It is a block diagram which shows the schematic structure of the mobile telephone to which the information terminal device which concerns on this invention is applied. It is a block diagram which shows the schematic structure of the mobile telephone to which the information terminal device which concerns on this invention is applied. It is a figure which shows the screen of the received image displayed on the display part of FIG. It is a block diagram which shows the schematic structure of the mobile telephone to which the information terminal device which concerns on this invention is applied. FIG. 12 is a diagram showing a screen of a reference captured image captured by the imaging unit in FIG. FIG. 12 is a diagram showing a screen of an actual captured image captured by the imaging unit in FIG. FIG. 12 is a diagram showing a screen of a reference received image displayed on the display unit of FIG. FIG. 12 is a diagram showing a screen of a received image after correction displayed on the display unit of FIG. It is a block diagram which shows the schematic structure of the mobile telephone to which the information terminal device which concerns on this invention is applied. FIG. 12 is a diagram showing a screen of an actual received image that is smaller than a reference received image displayed on the display unit of FIG. It is a block diagram which shows the schematic structure of the mobile telephone to which the information terminal device which concerns on this invention is applied. FIG. 19 is a diagram showing a screen of a received image after correction displayed on the display unit of FIG.

Explanation of symbols

10 Mobile phone
12 Imaging unit
14 Face detector
16 Communications department
18 Image controller
20 Display section
46 Display magnification control function
48 Facial feature extraction unit
50 Display image creation section
52 Character data storage

Claims (7)

Imaging means for imaging a user, communication means for transmitting an image taken to a communication partner apparatus, receiving an image from the communication partner apparatus, and display means for displaying at least the received image An information terminal device that communicates the captured images with each other, the information terminal device comprising:
A face detection unit that calculates a positional deviation amount of the face of the user from a reference point on the screen of the display unit from the captured image, and outputs the calculated positional deviation amount as positional information;
Image control means for shifting and controlling a received image matching the reference point received via the communication means based on the position information and outputting the controlled image to the display means Terminal device.   The information terminal device according to claim 1, wherein the image control unit sets a predetermined size for a face image to be displayed as a reference size, and sets a display magnification according to a ratio of the captured image size to the reference size. An information terminal device comprising: a magnification changing function block that generates the received image that has been changed and outputs the generated received image to the display means. Imaging means for imaging a user, communication means for transmitting an image taken to a communication partner apparatus, receiving an image from the communication partner apparatus, and display means for displaying at least the received image An information terminal device that communicates the captured images with each other, the information terminal device comprising:
A face detection unit that calculates a displacement amount from a reference point on the screen of the display unit of an image received from the communication partner device, and outputs the calculated displacement amount as position information;
And an image control means for controlling a shift of the received image with respect to the reference point based on the position information and outputting the controlled image to the display means. . Imaging means for imaging a user, communication means for transmitting an image taken to a communication partner apparatus, receiving an image from the communication partner apparatus, and display means for displaying at least the received image An information terminal device that communicates the captured images with each other, the information terminal device comprising:
A face detection unit that calculates a positional deviation amount of the face of the user from a reference point on the screen of the display unit from the captured image, and outputs the calculated positional deviation amount as positional information;
The reference that corrects and controls the amount of misalignment of the face in the captured image based on the position information, outputs the corrected captured image to the communication unit, and receives the corrected image based on the position information via the communication unit. An information terminal device comprising: an image control unit that controls a received image that coincides with a point by shifting and outputs the controlled image to the display unit. 5. The information terminal device according to claim 4, wherein the communication unit is connected to supply the received image to the face detection unit.
The face detection unit calculates a shift amount of the face position in the received image from the communication partner apparatus from a reference point on the screen of the display unit, and calculates the calculated shift amount of the face position of the user and Output the position displacement amount of the face in the received image as position information to the image control means,
The image control means performs control according to a shift of the received image with respect to the reference point received via the communication means based on the position information, and outputs the controlled image to the display means. Information terminal device. 6. The information terminal device according to claim 5, wherein the face detection unit sets a predetermined size for the face image in the captured image and the received image as each reference size, and the captured image with respect to the reference point and A displacement amount of a face image position in the received image is calculated, a ratio of the captured image to the reference point and a ratio of the face image in the received image to each reference size is calculated, and the calculated face position of the user Output the positional deviation amount and the ratio of the face in the received image and the ratio to the image control means as position information,
The image control means adjusts the ratio of the positional displacement amount and the reference size of the face with respect to the captured image based on the position information and outputs the ratio to the communication means, and based on the position information An information terminal device comprising a function of adjusting a ratio between the amount of positional deviation of the face and a reference size with respect to a received image and outputting the adjusted ratio to the display means. Imaging means for imaging a user;
Extracting means for extracting facial feature data included in the captured image;
Means for storing character data;
A communication means for transmitting a captured image to a communication partner device and receiving an image from the communication partner device;
Image creating means for creating an image based on the character data and the feature data of the face;
And at least display means for displaying the received image,
The extraction unit sets a predetermined size for the face image in the captured image and the received image as each reference size, and the captured image with respect to a reference point on the screen of the display unit and a face image in the received image Calculating a ratio of the captured image and the received image to the respective reference sizes of the face image, calculating the calculated positional shift amount of the user and the ratio, and the ratio Output the position displacement amount and the ratio of the face in the received image as position information to the image creating means,
The image creating means adjusts the ratio of the positional displacement amount and the reference size of the face with respect to the captured image based on the position information and outputs the ratio to the communication means, and the function based on the position information. A function of adjusting the ratio of the amount of positional deviation of the face with respect to the received image and a reference size, and outputting to the display means
An information terminal device that communicates the captured images with each other.

Images