JP2023099729A - Mobile information terminal - Google Patents

Abstract

To provide a technology that can realize a hands-free videophone with more suitable usability.SOLUTION: A mobile information terminal 1 has a videophone function and is equipped with a first camera (in-camera C1) including a wide-angle lens at a predetermined position (point PC1) on a front surface s1 having a display screen DP in a flat plate-shaped housing. When a first user (user A) makes a videophone call with another second user by using the videophone function, the housing is placed flat on a first surface (horizontal plane s0) of an object and a state in which the face of the first user is included in a range of an angle of view AV1 of the in-camera C1 is a first state. The mobile information terminal 1 detects a first area including the face of the first user from a wide-angle image taken by the in-camera C1, trims a first image corresponding to the first area, and creates a transmission image to be transmitted to another terminal on the basis of the first image, and transmits the transmission image to the other terminal.SELECTED DRAWING: Figure 2

Description

TECHNICAL FIELD The present invention relates to technology of a mobile information terminal such as a smart phone, and more particularly to a video phone function.

2. Description of the Related Art In recent years, mobile information terminals such as smartphones and tablet terminals have been miniaturized by high-density mounting in a single roughly flat plate-shaped housing, and have been equipped with various functions and advanced functions. These functions include a television reception function, a digital camera function, a videophone function, and the like. In particular, the videophone function of mobile information terminals is becoming easier to use, for example, through applications and services such as Skype (registered trademark).

In addition, as a digital camera function, the mobile information terminal has a camera (also called an in-camera) provided on the front of the main surface of the housing on the side where the display screen is located, and a camera provided on the back opposite to the front. It may have a camera (also called out-camera).

When a user makes a videophone call (sometimes referred to as a call) with a portable information terminal, the user uses the front in-camera, which is the side on which the user himself/herself can be photographed. The mobile information terminal displays the image of the other party received from the terminal of the other party on the videophone on the display screen, and transmits the image of the user himself/herself taken by the in-camera to the terminal of the other party. For this reason, the user normally holds the front of the mobile information terminal held in one hand, for example, in a position facing the user's face and line of sight, for example, in a position close to a state in which the housing is standing vertically. need to keep In this case, the user does not have both hands free.

When a user makes a videophone call using a portable information terminal, he or she may want to do so in a state in which both hands are free (sometimes referred to as "hands-free"). In this case, the user can place the housing of the mobile information terminal on a horizontal surface such as a desk, and face the display screen and the in-camera in a vertical direction. video calls are possible.

As prior art examples related to the mobile information terminal and the videophone function, Japanese Patent Application Laid-Open No. 2005-175777 (Patent Document 1) and Japanese Patent Application Laid-Open No. 2007-17596 (Patent Document 2) can be cited. Japanese Patent Application Laid-Open No. 2002-200002 describes that the mobile phone can improve the user's visibility when placed on a desk, and that the shape of the main body can be changed to support a hands-free videophone. Patent Document 2 describes a mobile terminal device that obtains information on a user's face based on an image captured by a camera of the main body, grasps the relative positional relationship between the orientation of the face and the orientation of the main body, It is described that the orientation of information to be displayed on the display screen is determined.

JP 2005-175777 A JP 2007-17596 A

When a user makes a videophone call using a portable information terminal, it may be inconvenient in a normal state where both hands are not free (sometimes referred to as non-hands-free). For example, the user cannot make a call while operating the PC with both hands, and it is difficult to make a call while showing some object such as a document to the other party.

In addition, when a user wants to realize a hands-free state in which both hands are free when making a videophone call using a portable information terminal, the user faces a housing that is placed flat on a horizontal surface such as a desk. It is necessary to take an unreasonable posture to achieve the positional relationship, which is not convenient. Alternatively, it can be realized by arranging the housing obliquely on a horizontal surface such as a desk using a fixing device, but this is inconvenient because the fixing device is required. Alternatively, a similar state can be achieved by using a mobile information terminal having a structure that allows the posture to be changed, such as a foldable type or a separate type. cannot be applied to

An object of the present invention is to improve the hands-free state of the videophone in relation to the technology of a portable information terminal having a videophone function, on the premise of having a configuration that has a generally flat plate-shaped housing and does not have a special deformable structure. To provide a technique that can be realized with suitable usability.

A representative embodiment of the present invention is a portable information terminal, characterized by having the following configuration. A mobile information terminal of one embodiment is a mobile information terminal having a videophone function, comprising a first camera including a wide-angle lens at a predetermined position on the front surface of a flat plate-shaped housing having a display screen, When a first user makes a videophone call with a second user on the other end using the videophone function, the housing is placed flat on the first surface of the object, and the first screen of the first camera is used. A first state is a state in which the face of the first user is included within a range of angles, and a first state including the face of the first user is obtained from a wide-angle image captured by the first camera in the first state. detecting a region, trimming a first image corresponding to the first region, and based on the first image, creating a transmission image to be transmitted to a partner terminal that is a mobile information terminal of the second user; The transmission image is transmitted to the remote terminal.

According to the representative embodiments of the present invention, a hands-free videophone can be realized with more suitable usability.

1 is a diagram showing configurations of a communication system and a videophone system including a mobile information terminal according to Embodiment 1 of the present invention; FIG. FIG. 10 is a diagram showing the positional relationship between the user and the terminal and the arrangement state of the terminal when the mobile information terminal of the first embodiment is used for a videophone call in a hands-free state; 1 is a diagram showing a configuration of a portable information terminal according to Embodiment 1; FIG. 2 is a diagram showing a software configuration in the portable information terminal of Embodiment 1; FIG. 2 is a diagram showing a configuration of a camera unit in the portable information terminal of Embodiment 1; FIG. 4 is a diagram showing a processing flow in the portable information terminal of Embodiment 1; FIG. 4 is a diagram showing an example of a display screen of the portable information terminal according to the first embodiment; FIG. FIG. 4 is a diagram showing an example of an image of a face taken from the front by a normal camera in the mobile information terminal of Embodiment 1; 4 is a diagram showing an example of a wide-angle image and face detection by an in-camera in the portable information terminal of Embodiment 1; FIG. 4 is a diagram showing an example of trimming from a wide-angle image in the portable information terminal of Embodiment 1; FIG. 4 is a diagram showing an example of a trimmed image (in the case of a rectangle) in the mobile information terminal of Embodiment 1; FIG. 4 is a diagram showing an example of a trimmed image in the portable information terminal of Embodiment 1; FIG. 4 is a diagram showing an example of a model and a coordinate system using a wide-angle lens in the portable information terminal of Embodiment 1; FIG. 4 is a diagram showing normal image conversion in the portable information terminal of Embodiment 1. FIG. FIG. 10 is a diagram showing trapezoidal correction in the portable information terminal according to the first embodiment; 4 is a diagram showing a first example of another usage state of the mobile information terminal of Embodiment 1. FIG. FIG. 10 is a diagram showing a second example of another usage state of the portable information terminal of Embodiment 1; FIG. 10 is a diagram showing a third example of another usage state of the mobile information terminal of Embodiment 1; FIG. 10 is a diagram showing a positional relationship and the like in a non-hands-free state in a mobile information terminal as a comparative example with respect to Embodiment 1 and in another videophone mode of Embodiment 1; FIG. 10 is a diagram showing the configuration of a camera unit in the portable information terminal according to Embodiment 2 of the present invention; FIG. 10 is a diagram showing the positional relationship between the user and the terminal in the portable information terminal of the second embodiment; FIG. 10 is a diagram showing a photographing process and an example of an image in a portable information terminal according to a modification of the embodiment; FIG. 10 is a diagram showing an example of an image including a specific object in a mobile information terminal according to a modification of the embodiment; FIG. 10 is a diagram showing an example of an image including faces of a plurality of users in a portable information terminal according to a modification of the embodiment; FIG. 10 is a diagram showing an image example of the partner image correcting function in the mobile information terminal of the modified example of the embodiment; FIG. 10 is a diagram showing an example of a state in which a hands-free videophone is realized by a mobile information terminal of a comparative example with respect to the embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In principle, the same parts are denoted by the same reference numerals in all the drawings for describing the embodiments, and repeated description thereof will be omitted. As directions for explanation, the X direction, the Y direction, and the Z direction are used. The X direction and the Y direction are two orthogonal directions forming a horizontal plane, and the Z direction is the vertical direction. In particular, the X direction is the left-right direction as seen from the user, and the Y direction is the front-back direction as seen from the user.

[Problems, etc.]
Supplementary explanation of issues, etc. FIG. 26 shows, as a comparative example, an example of a usage state when trying to achieve a hands-free state using a conventional mobile information terminal with a videophone function. (A) shows the first example, and (B) shows the second example. In (A) of FIG. 26, the user A has placed the housing of the portable information terminal 260 flat on a horizontal surface s0 such as a desk. A display screen and an in-camera 261 are provided on the front surface of the housing. The lens of the in-camera 261 is arranged at the position of the point p2. The in-camera 261 is a normal camera with a normal angle of view. A representative point of user A's face and eyes is indicated by point p1. The line of sight from the point p1 is vertically downward. User A takes a posture in which his or her face and eyes (point p1) face the display screen on the front surface of the housing and the in-camera 261 . In this state, user A can make a videophone call with both hands free. However, since the posture is unreasonable, the burden is placed on the body, and it is not convenient to use.

In FIG. 26B, user A fixes and arranges the housing of portable information terminal 260 on horizontal surface s0 such as a desk using fixture 262 so that the front surface is slanted. ing. The direction of the line of sight from the user A's face and eyes (point p1) is obliquely downward. The direction of the optical axis from the in-camera 261 (point p2) is obliquely upward (for example, an elevation angle of about 45 degrees). In this state, user A can make a videophone call with both hands free. However, such a state cannot be realized without using a fixing device 262 such as a stand, and the user A has to carry the fixing device 262. This lacks versatility and convenience.

A similar state can be achieved by using a portable information terminal having a structure that allows the posture to be changed instead of the fixture 262 . For example, in the technique disclosed in Patent Literature 1, by changing the shape of the foldable portable information terminal, the camera and display screen can be arranged so as to face the user's face. However, such a technique cannot be applied to the mobile information terminal 261 having a single, substantially flat plate-shaped housing, because it does not have a structure itself that can move due to deformation.

(Embodiment 1)
A portable information terminal according to Embodiment 1 of the present invention will be described with reference to FIGS. 1 to 19. FIG. In the mobile information terminal of the first embodiment, as shown in FIG. 2 described later, when the user makes a videophone call using the mobile information terminal (sometimes simply referred to as a terminal), the housing held by the user is used. It becomes a hands-free state without the need to hold the Also, when the user wants to make a hands-free videophone call, there is no need to consider or trouble arranging the housing (front face) of the mobile information terminal so that it faces the user's face. The portable information terminal of Embodiment 1 has a configuration such as a camera as described below in order to realize a suitable videophone in a hands-free state, and the positional relationship between the user's face and the terminal and the position of the terminal. Defines the placement state.

The portable information terminal of Embodiment 1 has a wide-angle camera having a wide angle of view as an in-camera (first camera) provided on the front surface of the housing. This in-camera has an optical axis perpendicular to the front surface of the housing, and a predetermined wide angle of view of, for example, about 180 degrees (at least an angle range from 30 degrees to 150 degrees) in a cross section formed by the front surface and the optical axis. (360 degrees as a horizontal angle of view).

When using the videophone in a hands-free state, the user lays down the generally flat plate-shaped housing flat on a generally horizontal surface such as a desk so that the front camera faces upward. Then, the positional relationship is such that the user's face is obliquely upward when viewed from the position of the in-camera of the terminal. When viewed from the user's eyes, the positional relationship is such that the display screen of the housing and the in-camera are positioned in a diagonally downward direction. In this state (first state), the direction in which the face is captured by the in-camera and the direction in which the user sees the display screen are approximately the same or sufficiently close. For example, these directions are elevation angles of about 45 degrees with respect to the horizontal plane (for example, angles within the angle range of 30 degrees to 60 degrees).

In the case of such a positional relationship and arrangement state, the user's face is captured in a part of the in-camera image (wide-angle image) (for example, an area within an angle range of 0 degrees to 90 degrees). there is Using the wide-angle image, the mobile information terminal can capture an area including the user's face. The portable information terminal detects an area including the user's face from the wide-angle image and trims (cuts out) it.

However, within the wide-angle image through the wide-angle lens, distortion (distortion) peculiar to the wide-angle lens occurs in the entire wide-angle image including the user's face. When a wide-angle image is transmitted to the other party's portable information terminal, the other user may have difficulty recognizing the user's face or the like from the wide-angle image. Therefore, the portable information terminal according to the first embodiment obtains a planarized image in which the distortion is eliminated or reduced by performing distortion correction processing on the area including the face of the wide-angle image. As a result, it is possible to obtain a transmission image that is visually easier to see and suitable for transmission.

The portable information terminal 1 creates a transmission image to be transmitted to the terminal of the other party of the videophone call from the image including the face of the user A after the distortion correction, and transmits the transmission image to the terminal of the other party. As described above, the mobile information terminal according to the first embodiment is convenient because the user can make videophone calls in a hands-free state.

[(1) Communication system and videophone system]
FIG. 1 shows the overall configuration of a communication system and videophone system including a mobile information terminal 1 according to the first embodiment. In the communication system and video phone system of FIG. It is connected. Videophone communication is performed between the mobile information terminal 1 of the first user and the mobile information terminal 2 of the second user. A base station 103 is a radio base station that constitutes the mobile communication network 101 . Mobile information terminals 1 and 2 are connected to mobile communication network 101 through base station 103 . The access point 104 is an access point device for wireless communication in a wireless LAN or the like. Mobile information terminals 1 and 2 are connected to the Internet 102 including a wireless LAN or the like through an access point 104 .

The mobile information terminals 1 and 2 are devices such as smartphones, and both have a videophone function. The mobile information terminal 1 of the first user is the mobile information terminal of the first embodiment, and has specific functions related to the videophone. User A is the first user of one caller who makes the videophone call, and user B is the other caller of the videophone call, the second user of the other party viewed from user A. Below, the mobile information terminal 1 of the first user will be mainly described. User B uses, for example, a conventional portable information terminal 2 with a videophone function. The mobile information terminal 2 of the second user may also have a unique videophone function like the mobile information terminal 1 of the first user. Note that a server or the like that provides services related to the videophone function may intervene on the mobile communication network 101 or the Internet 102 during videophone communication.

[(2) Overview of videophone usage]
FIG. 2 is a schematic diagram showing an outline, situation, and method of use of the portable information terminal 1 by user A during a videophone call between user A and user B in FIG. FIG. 2 shows the positional relationship between the user A's face and the mobile information terminal 1 and the layout of the terminals when the user A makes a hands-free videophone call. The outline of the use of the videophone is as follows.

(1) As shown in FIG. 2, when a user A makes a hands-free videophone call, the flat plate-shaped housing of the portable information terminal 1 is placed on a horizontal plane s0 (XY plane, The in-camera C1 on the front surface s1 is placed flat on the first surface) with the in-camera C1 facing vertically upward. The rear surface s2 of the housing is in contact with and hidden from the horizontal surface s0. The mobile information terminal 1 is mainly provided with a rectangular display screen DP on the front surface s1 of the vertically long housing, and a camera, operation buttons, a microphone, a speaker, etc. are provided in the frame area around the display screen DP. It is In this portable information terminal 1, an in-camera C1 (especially a wide-angle lens portion) is arranged at a position PC1 on the upper side of the frame area. When arranging the housing, the user A places the housing so that the in-camera C1 (position PC1) on the front surface s1 is positioned farther in the Y direction from the user A in the Y1 direction.

User A places his/her own face and eyes obliquely above the portable information terminal 1 . In other words, the display screen DP (point PD) of the mobile information terminal 1 is arranged diagonally below the user A's eyes (point P1). A representative point associated with user A's head, face, eyes, or the like is assumed to be point P1. An image of the other party (user B) and the like are displayed on the display screen DP (FIG. 7). The user A sees the other party's face image or the like on the display screen DP. A direction J1 indicates a line-of-sight direction in which the display screen DP (point PD) of the terminal is viewed from the user A's eyes (point P1). An angle θ1 indicates an elevation angle (0 degrees with respect to the horizontal plane s0) corresponding to the direction J1. The angle θ1 is an angle within the range of 30 degrees to 60 degrees, for example, about 45 degrees.

In this state, of course, the in-camera C1 of the terminal is in a state where the face and eyes of the user A can be photographed obliquely upward in the direction of the elevation angle. The optical axis of the in-camera C1 is in a direction perpendicular to the front surface s1 (vertically upward), which is shown as a direction J2. The angle of view AV1 of the in-camera C1 has a wide angle range centering on the optical axis. It has an angular range up to ANG2. For example, the first angle ANG1 is 30 degrees or less, and the second angle ANG2 is 150 degrees or more.

In this state, the in-camera C1 can photograph the face of the user A with a wide angle of view AV1. That is, in this state, within the angle of view AV1 of the in-camera C1, particularly within the angle range from the first angle ANG1 to 90 degrees, the angle of view AV2 corresponding to the range in which the face of the user A is photographed (face photographing angle of view) )It is included. Correspondingly, the face of the user A is shown in a part of the wide-angle image of the in-camera C1. A direction J3 indicates a direction in which the user A's face (point P1) is captured from the in-camera C1 (point PC1). An elevation angle corresponding to the direction J3 is indicated by an angle θ3. The angle θ3 is, for example, an angle slightly smaller than 45 degrees.

The line-of-sight direction J1 of the user A is sufficiently close to the face photographing direction J3 from the in-camera C1, and the angle difference AD1 between the angles θ1 and θ3 is sufficiently small. Therefore, the in-camera C1 can photograph the face of the user A in the direction J3 and the angle of view AV2 in which the user A's line of sight can be confirmed. It is preferable that the directional difference and the angular difference are as small as possible, because the state of the line of sight in the image becomes more natural. In the mobile information terminal 1 of Embodiment 1, the wide-angle image of the in-camera C1 is used, so even with the positional relationship shown in FIG.

(2) In the state shown in FIG. 2, user A can make a videophone call (call) while viewing the other party's image (FIG. 7) displayed on the display screen DP of the portable information terminal 1. FIG. Mobile information terminal 1 outputs the voice received from mobile information terminal 2 of the other party from a speaker. The mobile information terminal 1 transmits the voice of the user A input by the microphone to the mobile information terminal 2 of the other party.

The mobile information terminal 1 detects an area including the face of the user A from the wide-angle image captured by the in-camera C1, and trims the area. Using the trimmed image, mobile information terminal 1 creates a transmission image to be sent to mobile information terminal 2 of the other party. However, the wide-angle image captured by the in-camera C1, including the face of the user A, has distortion dependent on the wide-angle lens.

Therefore, the mobile information terminal 1 performs distortion correction processing on the trimmed image so as to eliminate or reduce the distortion. The portable information terminal 1 creates a transmission image to be transmitted to the other party's portable information terminal 2 and a monitor image for confirming the state of the user A's own face, etc. corresponding to the transmission image from the image after distortion correction. do. The portable information terminal 1 displays the monitor image corresponding to the transmitted image within the display screen DP (FIG. 7). In the monitor image (and the transmitted image), the line of sight of the user A is generally facing the front. User A can see and confirm the image of the other party (user B) and the monitor image of user A himself on the display screen DP of portable information terminal 1 . User A can also refuse transmission of the transmission image corresponding to the monitor image, if necessary.

The portable information terminal 1 transmits data for a videophone call, including user A's voice input with a microphone and the transmission image, to the other party's portable information terminal 2 . The mobile information terminal 2 of the other party outputs an image and voice regarding the user A based on the received data, and the user B can make a videophone call with the user A. FIG.

[(3) Portable information terminal]
FIG. 3 shows the configuration of the portable information terminal 1 according to the first embodiment. The mobile information terminal 1 includes a controller 10, a camera section 11, a ROM 14, a RAM 15, an external memory 16, a microphone 17, a speaker 18, a bus 19, a display section (touch panel) 21, a LAN communication section 22, a mobile network communication section 23, and an acceleration sensor. 31 and sensors 30 such as a gyro sensor 32, and the like.

The controller 10 controls the entire portable information terminal 1 and issues instructions to each unit. The controller 10 implements a videophone function 100 based on a videophone application. The controller 10 of the mobile information terminal 1 controls the videophone function 100 using each unit and each function. The controller 10 is composed of a microprocessor unit (MPU) or the like, and controls the entire mobile information terminal 1 according to a program in the ROM 14 . Each unit such as the controller 10 performs data transmission/reception with each unit in the portable information terminal 1 via a bus 19 (including a system bus).

The camera unit 11 includes an in-camera C<b>1 , a normal camera (out-camera) C<b>2 , a photographing processing unit 12 and a memory 13 . As shown in FIG. 2, an in-camera C1 is provided on the front surface s1 of the housing, and a normal camera C2 is provided on the rear surface s2 of the housing. The in-camera C1 is composed of known elements such as a wide-angle lens, a camera sensor (imaging device), and a drive circuit. The camera sensor is composed of, for example, a CCD or CMOS sensor. The normal camera C2 is generally composed of known elements such as a lens, a camera sensor, and a drive circuit. The normal camera C2 has a normal angle of view, which is narrower than the angle of view AV1 of the wide-angle lens of the in-camera C1. The optical axis of the normal camera C2 is in the opposite direction to the optical axis of the in-camera C1. The videophone function in the first embodiment does not normally use the camera C2.

The shooting processing unit 12 is a part that performs shooting processing and image processing using a camera under the control of the controller 10 . In Embodiment 1, the imaging processing unit 12 is implemented as a circuit such as an LSI, for example, as a part separate from the controller 10 . Without being limited to this, the photographing processing unit 12 may be integrally implemented by program processing or the like of the controller 10 in part or in its entirety. All or part of the functions such as the face detection function 201 may be implemented by software program processing, or may be implemented by hardware circuits or the like for speeding up. The memory 13 is a memory for storing image data and the like related to photographing processing.

The photographing processing unit 12 includes, as well-known functions and processing units, an autofocus function, a scaling function, a codec unit, an image quality improvement processing unit, an angle/rotation correction unit, and the like. The autofocus function is a function that automatically focuses the camera on an object to be photographed. The enlargement/reduction function is a function for enlarging or reducing an object of an image. The codec unit is a processing unit that compresses and decompresses captured images and moving images. The image quality improvement processing unit is a processing unit that improves the image quality of the captured image, such as noise removal. The angle/rotation correction unit is a processing unit that performs angle correction and rotation correction from the captured image.

The microphone 17 collects sounds around the terminal, including the voice of the user A, to obtain sound data. The speaker 18 outputs audio including videophone audio from the portable information terminal 2 of the other party (user B).

The display unit 21 includes the display screen DP shown in FIG. 2, and is particularly a touch panel such as a liquid crystal touch panel, and allows the user to perform a touch input operation. A captured image and various other information are displayed on the display screen DP.

The LAN communication unit 22 performs communication processing corresponding to communication on the Internet 102 including wireless LAN communication with the access point 104 in FIG. The mobile network communication unit 23 performs communication processing corresponding to communication on the mobile network 101 including wireless communication with the base station 103 in FIG.

The sensors 30 include known sensor devices such as an acceleration sensor 31, a gyro sensor 32, a GPS receiver (not shown), a proximity sensor, an illuminance sensor, and a temperature sensor. The controller 10 uses detection information from the sensors 30 to detect the orientation, movement, and the like of the mobile information terminal 1 . Using the sensors 30, the controller 10 can also determine whether or not the mobile information terminal 1 is being held by the user A, or whether or not the mobile information terminal 1 is placed flat on the horizontal surface s0 as shown in FIG. be.

The photographing processing unit 12 has a face detection function 201, a trimming function 202, and a distortion correction function 203 as functions realized by programs, circuits, and the like. The outline of the shooting processing and functions of the shooting processing unit 12 is as follows. The imaging processing unit 12 inputs a wide-angle image, which is an image captured by the in-camera C1, in a camera mode in which an image is captured using the in-camera C1. In Embodiment 1, the in-camera C1 can shoot a moving image (a plurality of time-series image frames), and the shooting processing unit 11 processes the moving image. The image captured by the in-camera C1 is not limited to this, and may be a still image captured at a predetermined timing.

The imaging processing unit 12 detects an area including the face of the user A from the wide-angle image using the face detection function 201 . The photographing processing unit 12 uses the trimming function 202 to trim an area including the user A's face from the wide-angle image. The imaging processing unit 12 performs distortion correction processing on the trimmed image using the distortion correction function 203 . The imaging processing unit 12 stores the distortion-corrected image in the memory 13 . The photographing processing unit 12 (or the controller 10) creates a transmission image for transmission to the other party's mobile information terminal 2 and a monitor image for self-confirmation from the corrected image.

The controller 10 (videophone function 100) of the portable information terminal 1 creates videophone data in which the user A's transmission image and the user A's voice data input from the microphone 17 are put together. The controller 10 uses the LAN communication unit 22, the mobile network communication unit 23, or the like to transmit the data to the portable information terminal 2 of the other party. The mobile information terminal 2 of the other party receives the data, displays the transmitted image on the display screen, and outputs sound.

The controller 10 (videophone function 100) receives videophone data (including the partner's image and voice) from the other party's portable information terminal 2 via the LAN communication unit 22 or the mobile network communication unit 23. FIG. The controller 10 displays the other party's image in the received data on the display screen DP, and also displays the user A's monitor image. The controller 10 outputs the other party's voice from the speaker 18 .

[(4) Software configuration]
FIG. 4 shows the software configuration of the portable information terminal 1. As shown in FIG. The ROM 14 stores a basic operation program 14a such as an OS and middleware, and other application programs. For the ROM 14, for example, a rewritable ROM such as EEPROM or flash ROM is used. Through communication or the like, the program in the ROM 14 can be updated as appropriate, and version upgrades, function extensions, and the like are possible. The ROM 14 and the like may be integrated with the controller 10 .

The RAM 15 is used as a work area during execution of the basic operation program 14a, videophone application program 16b, and the like. The RAM 15 also has a temporary storage area 15c that temporarily holds data and information as required during execution of various programs. The controller 10 (MPU) develops the basic operation program 14a of the ROM 14 in the RAM 15 and executes processing according to the program. Thus, the RAM 15 is configured with a basic operation executing section 15a. Similarly, a videophone processing execution unit 15b is configured in the RAM 15 in accordance with the processing of the videophone application program 16b. The temporary storage area 15c stores data for processing related to the videophone function 100, and also stores information such as the position and attitude of the portable information terminal 1, for example.

The external memory 16 stores programs such as a shooting program 16a and a videophone application program 16b. It also has a data storage area 16c. The external memory 16 is composed of a non-volatile storage device that retains data even when power is not supplied. For example, a flash ROM, an SSD, or the like is used. The data storage area 16c also stores, for example, settings for the functions and operations of the mobile information terminal 1 . Various programs may be stored in the ROM 14 or other non-volatile storage devices. The mobile information terminal 1 may acquire programs and information from an external server device or the like.

The shooting program 16 a implements shooting control processing for the shooting processing unit 12 of the camera unit 11 . This photographing control processing includes general camera photographing control processing that is not limited to videophones, and photographing control processing for videophones. The photographing program 16a is expanded in the RAM 15 or the photographing processing section 12, and constitutes an execution section.

The controller 10 (MPU) instructs the photographing processing unit 12 about the camera mode, the start and end of photographing of moving images by the camera, detailed photographing settings (for example, focus and exposure), and the like. The camera mode indicates which of a plurality of cameras is used for photographing.

The videophone processing execution unit 15b based on the videophone application program 16b performs processing corresponding to the videophone function 100. FIG. When the videophone function 100 is realized, the controller 10 (MPU) performs control processing for each function of the shooting processing section 12 and control processing for each related section.

[(5) Camera unit, shooting processing unit]
FIG. 5 shows detailed configurations of the camera section 11, the photographing processing section 12, and the memory 13. As shown in FIG. The face detection function 201 of the imaging processing unit 12 includes a personal recognition function 201B. The distortion correction function 203 performs normal image conversion processing 203A, keystone correction processing 203B, aberration correction processing 203C, and the like as processing. The memory 13 stores data such as a registered image D10, a corrected image D11, a transmission image D12, a monitor image D13, and the like. The memory 13 temporarily holds captured image data and is also used as a work area for processing of each function. A memory 13 may be provided in the imaging processing unit 12 . The outline of the processing of the photographing processing unit 12 is as follows.

(1) First, a wide-angle image (data D1) shot through the in-camera C1 is input to the face detection function 201 of the shooting processing unit 12 . The face detection function 201 detects an area including the face of the user A from the wide-angle image data D1 based on image processing. The face detection function 201, for example, extracts a feature point group from within a wide-angle image, detects eyes, ears, nose, mouth, etc., and also detects the contours of the face and head based on differences in pixel color and brightness. to detect As a result, the face area B1 and the like can be detected as shown in FIG. 9, which will be described later. The face detection function 201 outputs a wide-angle image and detection result data D2.

Further, the individual recognition function 201B recognizes whether or not the face image is that of a specific user A from the face image. The photographing processing unit 12 applies subsequent processing only when, for example, the face of a specific user A is detected.

(2) Next, the trimming function 202 trims the trimming area corresponding to the area including the detected face based on the wide-angle image data D2 to obtain a trimmed image. The trimming function 202 outputs data D3 such as the trimmed image. As a method of trimming, for example, a region of a predetermined shape and size is trimmed with reference to the center point (point P1) in the detected face region. The trimming area may be only the face area, the head area, or the area including the head and its periphery. The type and size of the trimming area can be changed using the user setting function.

(3) Next, based on the data D3, the distortion correction function 203 performs processing for correcting the distortion caused by the wide-angle lens in the trimmed image so that the flat image has a normal image without distortion. The distortion correction function 203 first performs normal image conversion processing 203A (FIG. 14) on the trimmed image. As a result, a flat image (planarized image) having a normal image without distortion is obtained, and data D4 including the flattened image is output.

The distortion correction function 203 next performs keystone correction processing 203B (FIG. 15) on the data D4 of the planarized image. As a result, an image is obtained in which the trapezoidal image content is transformed into a rectangular image content, and data D5 thereof is output. Trapezoidal transformation causes the image content to have a more pleasing appearance.

Next, in the aberration correction processing 203C, the distortion correction function 203 performs known processing for correcting various types of aberration caused by the characteristics of the lens system, other than the wide-angle distortion, on the data D5. As a result, a post-correction image D11 is obtained. For example, when the lens system is fixed, the lens system correction parameters D14 are stored in the memory 13, for example, in advance at the time of product shipment. The lens system correction parameters D14 are setting information for aberration correction, initial values, etc., and may be set variable. The aberration correction processing 203C refers to the lens system correction parameter D14.

Normally, in the state of the image after the normal image conversion processing 203A, the state of the face of the user A is sufficiently comfortable to the eye (at least in a state that can be used for a videophone). Therefore, the trapezoidal transformation processing 203B and the aberration correction processing 203C may be omitted. Also, the processing performed by the distortion correction function 203 does not necessarily have to be performed in this order, and may be performed in any order. Also, depending on the conditions, control may be performed so that specific processing is not performed.

(4) The image D11 after correction by the distortion correction function 203 is stored in the memory 13. FIG. In the post-correction image D11, the distortion and lens aberration caused by the wide-angle image are eliminated or reduced, and the image is of a level that allows the user to recognize the face and the like with little discomfort. The photographing processing unit 12 or the controller 10 (videophone function 100) uses the post-correction image D11 to create a transmission image D12 for transmission to the other party (user B) and a monitor image D13 for self-confirmation. The controller 10 uses the transmitted image D<b>12 and the voice input by the microphone 17 to create data for videophone.

The transmission image D12 is, for example, an image that has been appropriately cut, enlarged/reduced, etc. so as to match the image size (display screen size, etc.) requested by the mobile information terminal 2 of the other party. The monitor image D13 is an image that has been appropriately cut or enlarged/reduced so as to match the size of the monitor image display area (area R2 in FIG. 7) in the display screen DP.

(5) The controller 10 displays the monitor image D13 in an area within the display screen DP. When the result of confirmation by user A on monitor image D13 is affirmative (transmission permitted), controller 10 transmits data including transmission image D12 to portable information terminal 2 of the other party via the communication unit.

(6) If the face detection function 201 fails to detect the face of the user A, or if the user A confirms the monitor image D13 and receives a transmission refusal instruction, the photographing processing unit 12 , a transmission image D12 is created using the registration image D10. Registered image D10 includes user A's face image.

The photographing processing unit 12 similarly repeats the above processing for each photographed image at a predetermined time interval. At that time, if the face of user A cannot be completely captured from the image at a certain point in time, an alternative transmission image D12 may be created using the last detected image in the past or the face image of the registered image D10. good.

[(6) Processing flow]
FIG. 6 shows a processing flow of the videophone function 100 in the portable information terminal 1. As shown in FIG. The flow of FIG. 6 has steps S1 to S13. The steps will be described below.

(S1) First, in S1, when user A makes a videophone call (when he or she makes a call to the other party or when the other party receives an incoming call), the controller 10 (videophone function 100) of the mobile information terminal 1 automatically The control state of the machine is shifted to videophone mode. Specifically, for example, when user A wants to make a videophone call with a partner (user B), he calls the other party's telephone number. Along with this, the videophone application program 16b (videophone processing execution unit 15b) is started. In the videophone mode, the controller 10 concurrently controls the photographing of the in-camera C1, the voice input of the microphone 17, the voice output of the speaker 18, the display of the display unit 21, various communications, and the like.

In addition, the mobile information terminal 1 shifts to the videophone mode in response to the user A's operation of selecting a voice call (non-videophone) or a videophone. For example, the portable information terminal 1 displays a selection button for voice call or videophone on the display screen DP, and shifts to the corresponding mode according to the selection operation. Moreover, since the portable information terminal 1 uses the in-camera C1 in the videophone mode, the camera mode of the camera section 11 is set to the mode using the in-camera C1.

Furthermore, in Embodiment 1, as details of the videophone mode, two types of a normal mode (non-hands-free mode) and a hands-free mode are provided. The normal mode (non-hands-free mode) is the first mode corresponding to the state shown in FIG. Hands-free mode is the second mode corresponding to the state shown in FIG. The portable information terminal 1 selects one of these modes in accordance with a predetermined instruction operation by the user A or automatic terminal state grasping using the sensors 30 . For example, the mobile information terminal 1 may display selection buttons for the normal mode and the hands-free mode on the display screen DP, and shift to the corresponding mode according to the selection operation. Alternatively, the mobile information terminal 1 can grasp the state such as whether the user A is holding the housing by hand or laying it flat on the horizontal plane s0 from the detection information of the acceleration sensor 31, etc., and automatically mode may be determined explicitly.

In this example, user A makes a videophone call in a hands-free state (corresponding hands-free mode) as shown in FIG. User A puts the housing of portable information terminal 1 in the state shown in FIG. 2, and portable information terminal 1 selects the hands-free mode. It should be noted that in other embodiments, it is not necessary to distinguish between the two types of modes.

(S2) The mobile information terminal 1 recognizes the terminal state, sets the camera section 11 to a mode (in-camera mode) in which the in-camera C1 is used in the videophone mode, and starts photographing. The shooting processing unit 12 inputs a moving image from the in-camera C1.

(S3) The photographing processing unit 12 of the mobile information terminal 1 uses the face detection function 201 to detect an area including the face of the user A (for example, area B1 in FIG. 9) from the wide-angle image captured by the in-camera C1.

(S4) The photographing processing unit 12 uses the trimming function 202 to trim a predetermined region including the face from the region detected in S3 as a trimming region (for example, trimming region TRM1 in FIG. 10), and produces a trimmed image (for example, FIG. 12 image GT1) is obtained.

(S5) The photographing processing unit 12 uses the distortion correction function 203 to apply distortion correction processing (normal image conversion processing 203A) to the trimmed image obtained in S4. Further, the distortion correction function 203 performs the above-described trapezoidal correction processing 203B and aberration correction processing 203C. As a result, a corrected image D11 (for example, the image GP1 in FIG. 12) is obtained.

(S6) The photographing processing unit 12 (or the controller 10) creates a transmission image D12 and a monitor image D13 (for example, images GP11 and GP12 in FIG. 12) using the corrected image D11 obtained in S5.

(S7) The controller 10 displays the image received from the portable information terminal 2 of the other party (user B) in the area (area R1 in FIG. 7) within the display screen DP. Further, the controller 10 displays the monitor image D13 of the user A in the area (area R2 in FIG. 7) within the display screen DP.

(S8) The controller 10 confirms with the user A whether or not the state of the face of the monitor image D13 may be used as the corresponding transmission image D12 (sometimes referred to as transmission confirmation). For example, the display screen DP may display transmission confirmation information (for example, "Can I send the image?") and operation buttons (for example, a transmission permission button and a transmission refusal button). User A looks at the monitor image D13 or the like and determines whether or not the content of the image may be transmitted. For example, user A presses a transmission permission button or transmission refusal button on the display screen DP. If the transmission is permitted in S8 (Y), the process proceeds to S10, and if the transmission is rejected (N), the process proceeds to S9. When the user A looks at the monitor image D13 and feels uncomfortable with the condition of his/her eyes, for example, he/she can select to reject the transmission.

(S9) The controller 10 creates a substitute transmission image D12 using the registered image D10. Further, at that time, the controller 10 may confirm with the user A on the display screen DP whether or not the registered image D10 may be used as the alternative transmission image D12. For example, in the monitor image display area R2 in FIG. 7, the face image of the registered image D10 is displayed, and confirmation information and operation buttons are displayed.

(S10) The controller 10 transmits the videophone format data including the transmission image D12 and the like to the mobile information terminal 2 of the other party via the communication unit.

(S11) The controller 10 processes videophone calls between users A and B (including voice input and output, image display, etc.). Note that the voice data of the caller may be constantly transmitted and received separately from the image, as in the case of a normal telephone call.

(S12) The controller 10 confirms whether the videophone call has ended. For example, user A presses the end button when ending the videophone call. Alternatively, mobile information terminal 1 receives information indicating the end of communication from mobile information terminal 2 of the other party. If the videophone call is terminated (Y), the process proceeds to S13, and if continued (N), the process returns to S2. From S2, the processing for each time point is similarly repeated in a loop. User A's face is automatically tracked by the loop. Each process in the loop is streamlined so that the same process is not repeated as much as possible. For example, in the face detection process of S3, the face area once detected at a certain time is automatically tracked by motion detection or the like at a later time.

(S13) The controller 10 performs videophone mode termination processing, and terminates activation (execution) of the videophone application. The end processing includes resetting settings (for example, the number of retries) related to the videophone, erasing image data, and the like.

Supplements and modifications of the above processing flow are as follows. In the above processing flow, basically, each image of the moving image of the in-camera C1 is similarly processed in a loop. The transmission confirmation using the monitor image D13 in S8 is performed once, for example, at the start of the videophone call, that is, using the image of the first period of the moving image. If the transmission is confirmed to be permitted, the transmission image D12 created at each point in time is automatically transmitted during the subsequent videophone call. Not limited to this, the confirmation of transmission may be performed at regular intervals during the videophone call, may be performed in response to a predetermined user operation, or may not be performed at all. In the user setting function of the videophone application, it is possible to set whether or not to confirm transmission, timing, and the like. If the transmission confirmation is not set, the creation of the monitor image D13 in S6, the display of the monitor image D13 in S7, the transmission confirmation in S8, etc. can be omitted. to send.

Also, if the transmission is permitted by the transmission confirmation at the beginning of the videophone call, the same image as the transmission image D12 created first may continue to be used until the end of the videophone call or until the next transmission confirmation. good.

In the face detection process of S3, for example, when a face area cannot be detected from a wide-angle image at a certain point in time, the face detection process may be retried using an image at another point in time according to a preset number of retries. . If the face region cannot be detected or tracked, the last detected face image or the registered image D10 may be used as an alternative. In addition, the mobile information terminal 1 displays to the user A on the display screen DP that the face cannot be detected, etc., and confirms whether or not the registered image D10 is used as the transmission image D12 instead. good too.

In addition, when the user A instructs to reject the transmission in the transmission confirmation of S8, the mobile information terminal 1 may immediately use the registered image of S9. The process may be retried up to a predetermined number of retries by returning to steps such as the above. Settings such as the number of retries may be set as the default settings of the videophone application, or may be changed by user settings. Depending on the user's needs and operations, it is possible to use the user setting function of the videophone application to set whether or not to use various functions, including confirmation of transmission and creation of transmission images using registered images, and operation details. is. Depending on user settings, it is possible to use real-time camera images or only registered images D10 throughout the video call.

[(7) Personal digital assistant - display screen]
FIG. 7 shows the configuration of the display screen DP and the like on the XY plane when the front surface s1 of the portable information terminal 1 is viewed in plan during a videophone call. Of the main surfaces (front surface s1, rear surface s2) of the flat plate-shaped housing of the portable information terminal 1, the front surface s1 side having the display screen DP is provided with an in-camera C1 including a wide-angle lens portion. In the vertically long rectangular area of the front surface s1, the in-camera C1 is provided at, for example, the central position (point PC1) of the upper side of the frame area outside the area of the main display screen DP.

Based on the control of the videophone application and the image data received from the portable information terminal 2 of user B, an image (partner Image) g1 is displayed.

Along with the display of the image g1 in the area R1 within the display screen DP, a monitor image g2 including the face of the user A, which is created based on the image of the in-camera C1, is displayed in a predetermined partial area R2. This monitor image g2 is provided so that the user A can check the state of his/her own image to be transmitted to the other party during the videophone call. The function of displaying the monitor image g2 is not essential, but if it is displayed, the usability can be enhanced. User A can check the state of his/her own face, etc. by looking at this monitor image g2, and can refuse transmission of the transmission image D12 corresponding to this monitor image g2, if necessary.

In the display example of FIG. 7A, the other party's image g1 is displayed in the main area R1 corresponding to almost the entire display screen DP, and in the area R1, for example, the upper right corner near the in-camera C1 is displayed. A superimposed area R2 is provided at the position, and the monitor image g2 is displayed in the area R2.

In another display example of FIG. 7B, the main area R1 in which the image g1 of the other party is displayed in the display screen DA is arranged at a position closer to the in-camera C1 and closer to the upper side of the area R1. A divided area R2 is provided on the lower side, and the monitor image g2 is displayed in the area R2. Various display methods are possible without being limited to these, and can be changed by the user setting function.

In addition, in the display example of FIG. 7, the monitor image g2 in the area R2 is smaller in size than the other party's image g1 in the area R1. The size of the monitor image g2 in the area R2 can also be changed. It is also possible to enlarge and display only the monitor image g2 according to a touch operation on the display screen DP.

[(8) Image example, processing example]
8 to 12 show image examples and processing examples for face detection, trimming, and distortion correction based on the wide-angle image of the in-camera C1 of the portable information terminal 1. FIG.

(1) First, for comparative explanation, FIG. 8 shows an example of an image (normal image) when the user A's face is photographed from the front by the normal camera C2. This normal image shows a case of a square size. A central point or a representative point on the face or head of user A is defined as point P1. The point P1 may be the middle point between both eyes. Schematically, a face area A1, a head area A2 and an area A3 are shown. The face area A1 is an area that includes a face (including eyes, nose, mouth, ears, skin, etc.). The head area A2 is wider than the face area A1 and is an area that includes the head (including hair and the like). The area A3 is wider than the face area A1 and the head area A2, and is an area that includes a peripheral area outside the face or head to some extent. The area A3 may be, for example, an area up to a predetermined distance from the point P1, or may be an area taken according to the ratio of the face area A1 or the like in the area A3. The shape of each region is not limited to a rectangle, and may be an ellipse or the like. In the case of an image captured in a positional relationship in which the camera is viewed obliquely upward as shown in FIG. 2, the actual image content is slightly trapezoidal as shown in FIG. 15B.

(2) FIG. 9 schematically shows an overview of a wide-angle image G1 captured by the in-camera C1 and face area detection. This wide-angle image G1 has a circular area. A point PG1 indicates the center point of the wide-angle image G1 and corresponds to the direction J2 of the optical axis. Position coordinates in the wide-angle image G1 are indicated by (x, y). A region (face region) B1 indicated by a dashed frame schematically indicates a rectangular region corresponding to the face region A1. Similarly, an area (head area) B2 indicates a rectangular area corresponding to the head area A2, and an area B3 indicates a rectangular area corresponding to the area A3. Region B4 is also larger than region B3 and illustrates the case of taking a sufficiently large rectangular region for processing. Each region is shown as a rectangle, but the shape is not limited to this, and may be a shape that matches the coordinate system of the wide-angle image.

In the wide-angle image G1 of FIG. 9, based on the positional relationship shown in FIG. 2, the user's The face of A is shown. Thus, the wide-angle lens-dependent distortion occurs in the entire wide-angle image G1 captured through the wide-angle lens of the in-camera C1, including the face. Within the wide-angle image G1, distortion may be greater at the outer peripheral position than at the center (point PG1).

The face detection function 201 detects an area including a face from within the wide-angle image G1. The face detection function 201 detects, for example, the face area B1 or the head area B2.

(3) FIG. 10 shows the wide-angle image G1 of FIG. 9 superimposed on the (x, y) plane of the original images of FIGS. 13 and 14 to be described later. The trimming function 202 sets trimming regions corresponding to the face region B1, region B3, etc. from the wide-angle image G1 to obtain a trimming image. In the example of FIG. 10, from the wide-angle image G1, a shield-shaped region corresponding to the coordinate system of the (x, y) plane of the original image, which corresponds to the region B3, is set as the trimming region TRM1 (broken line frame). . The shield shape of the trimming region TRM1 is, for example, a shape in which the width in the x direction decreases from the center to the outer periphery in the y direction.

(4) Also, FIG. 11 shows an example of a trimmed image in the case of trimming the wide-angle image G1 of FIG. 9 as a rectangular (right-angled quadrilateral) trimming area for comparative explanation. A region 111 indicates a rectangular region including the head and its surroundings. Region 112 shows an example of a rectangular region that is larger than region 111 for processing. In an area 111, the horizontal width H1 of the face area, etc., and the overall horizontal width H2 are schematically shown. Regarding the determination of the size of the area 111, the overall width H2 is set so as to have a predetermined ratio (H1/H2) to the horizontal width H1 of the face area or the like. For example, H1/H2=1/2 or 2/3. This ratio is user configurable. Alternatively, the sizes of the areas 111 and 112 may be determined by taking predetermined distances K1 and K2 in the horizontal and vertical directions from the center point P1 of the face area.

(5) FIG. 12 shows an example of a trimming image and distortion correction. (A) of FIG. 12 shows a trimming image GT1 corresponding to the trimming region TRM1 of FIG. Also, an example in which the trimming region TRM2 and the trimming image GT2 are larger than the trimming region TRM1 and the trimming image GT1 for processing is shown. On the (x, y) plane of the original image, the trimming area TRM1 has a size that includes the periphery of the user A's face (for example, a size that includes the upper half of the body).

(B) of FIG. 12 shows a flattened image GP1 that is an image resulting from distortion correction on the trimmed image GT1 of (A). The distortion correction function 203 performs normal image conversion processing 203A (FIG. 14) on the trimming image GT1. As a result, a planarized image GP1 with almost no distortion is obtained. The planarized image GP1 has a rectangular (rectangular) plane PL1. A plane PL2 indicates a plane of a flattened image obtained in the same manner as in the case of the trimmed image GT2. In particular, the image GP11 and the image GP12 show an example of extracting a part of the planarized image GP1. Image GP11 corresponds to the face region, and image GP12 corresponds to the region including the head region and its periphery.

The imaging processing unit 12 acquires the flattened image GP1 as the post-correction image D11. Furthermore, the imaging processing unit 12 may extract a part of the image GP11 and the like from the flattened image GP1 and process it appropriately to create the monitor image D13.

The imaging processing unit 12 uses the distortion correction function 203 to perform a normal image conversion process 203A from the state of the wide-angle image G1 (original image) to a state without distortion. Here, when performing the calculation of the normal image conversion processing 203A on the entire circular area of the original image, there is concern that the amount of calculation will increase. Therefore, in the first embodiment, as an example of processing, as described above, the photographing processing unit 12 performs limited normal image conversion on an image obtained by trimming a part of the wide-angle image G1 (original image). Calculations such as processing 203A are performed.

[(9) Distortion correction]
A distortion correction method for correcting a distorted wide-angle image captured by the in-camera C1 into a distortion-free planarized image will be described with reference to FIGS. 13 and 14. FIG.

FIG. 13 shows a model of normal image conversion and a coordinate system when the wide-angle lens of the in-camera C1 is used. A hemispherical surface 500 corresponding to a wide-angle lens, a planar imaging surface 501 of a camera sensor, a planarized image 502, and the like are shown. The imaging plane 501 is also shown as the (x,y) plane of the original image. A planarized image 502 corresponds to a captured image of the target object and is shown as a (u, v) plane. A wide-angle lens has a shape close to a sphere although the radius and the like vary depending on the angle of view and the like, and is shown as a hemispherical surface 500 . The imaging surface 501 is arranged at a position perpendicular to the axis of the imaging range of the hemispherical surface 500 in the Z direction. The origin O, radius R, and three-dimensional coordinates (X, Y, Z) of the hemispherical surface 500 and imaging surface 501 are shown. The bottom plane of the spherical coordinate system (hemispherical surface 500) corresponds to the imaging surface 501 on the camera sensor and is shown as the (x, y) plane of the original image.

The mobile information terminal 1 uses the model of FIG. , the distance, angle, range, etc. from the center position can be determined. Therefore, the photographing processing unit 12 converts the distorted face image into a flattened image in which the distortion is eliminated or reduced by performing distortion correction processing based on the model of FIG. 13 (normal image transformation of FIG. 14). be able to.

As shown in FIG. 2 described above, the angle of view AV1 of the wide-angle lens of the in-camera C1 is as large as about 180 degrees. An object (for example, a face) is photographed in a nearly spherical shape (hemispherical surface 500 ) corresponding to a wide-angle lens, and a transmitted optical image is received by a camera sensor having a plane indicated by an imaging surface 501 . Then, the image (wide-angle image) on the camera sensor is distorted as shown on the left side of FIG. The relationship (indicated by angle β) between the front angle of the camera sensor (Z direction) and the captured image direction (n direction) becomes tighter toward the outer periphery than toward the center.

The left side of FIG. 14 conceptually shows the distortion in the (x, y) plane of the original image. In the (x, y) plane, the unit area of the coordinate system is not a rectangular rectangle (eg strain amounts δu, δv). The right side of FIG. 14 shows the (u, v) plane as a flattened image 502 . This (u, v) plane is an image to be extracted as an image without distortion. On the (u, v) plane, the unit area of the coordinate system is a rectangular rectangle (eg Δu, Δv).

The specifications regarding the focal length and lens shape of the wide-angle lens are known in advance. Therefore, it is possible to easily perform coordinate conversion from the spherical image (original image) to a planar image (referred to as a planarized image). As this coordinate transformation, normal image transformation as shown in FIG. 14 can be applied. This normal image conversion converts an image with distortion into an image without distortion as seen by the human eye, and is used for distortion correction of a fisheye lens, for example. Using this normal image conversion, the pixels at each point position on the (x, y) plane of the original image on the left side of FIG. do. Details of the conversion are below.

As shown in FIG. 13, let α be the azimuth angle, β be the zenith angle, and φ be the rotation angle of the extracted plane (planarized image 502) with respect to the shooting direction n of the target object. Let m be the magnification of the lens. Then, the following formulas 1 and 2 are obtained from the known normal image conversion formulas. Also, A, B, C, and D in the formula are formulas 3 to 6.
Formula 1: x=R(uA−vB+mR sinβsinα)/√(u ² +v ² +m ² R ² )
Equation 2: y=R(uC−vD+mR sinβcosα)/√(u ² +v ² +m ² R ² )
Equation 3: A = cosφcosα−sinφsinαcosβ
Equation 4: B = sin φ cos α + cos φ sin α cos β
Equation 5: C=cosφsinα+sinφcosαcosβ
Equation 6: D=sinφsinα−cosφcosαcosβ

By calculation according to the above formula, each pixel of the original image can be transformed into each pixel of the flattened image to eliminate the distortion. The amount of distortion δu, δv at each pixel position on the (x, y) plane of the original image is equivalent to a square unit area on the (u, v) plane of the flattened image, where there is no distortion (Δu, Δv) be made.

[(10) Keystone correction]
FIG. 15 shows an image example and the like for the trapezoidal correction processing 203B. As shown in FIG. 15A, when the portable information terminal 1 is placed flat on the horizontal plane s0, the face of the user A is photographed in an obliquely upward direction J3 as viewed from the in-camera C1. Therefore, at the angle of view AV2 of the photographed face image, the lower part of the face or head in the Z direction (such as the chin) is closer in distance than the upper part in the Z direction. The distance DST1 is the distance between the position PC1 of the in-camera C1 and the upper part of the head, and the distance DST2 is the distance between the position PC1 of the in-camera C1 and the lower part of the head. DST1>DST2.

Therefore, in the captured image (wide-angle image), as shown in FIG. 15B, the face area of the user A and the like have trapezoidal image contents (referred to as a trapezoidal image). In the trapezoidal image 151, the upper side in the Z direction is smaller than the lower side. A trapezoidal image 151 schematically shows the shape of the wide-angle image without distortion after conversion to the normal image. In the trapezoidal image 151, for example, the parietal side is relatively small and the chin side is relatively large.

If the transmission image D12 is configured using this trapezoidal image, the other party (user B) may feel a sense of discomfort when viewing the transmission image D12. Therefore, the keystone correction process 203B is used to create a more suitable transmission image D12. The mobile information terminal 1 uses the image of the in-camera C1 and sensors to determine the range of the elevation angle (angle θ1) when a point P1 at a representative position of the face of the user A is viewed from the position PC1 of the in-camera C1. can be determined using The photographing processing unit 12 performs keystone correction processing 203B based on the determined information.

FIG. 15C shows an image 152 resulting from trapezoidal correction from (B). This image 152 is a right-angled square image, and the upper side and the lower side have the same length. By creating the transmission image D12 using this image 152, the image appears as if the user A's face is viewed from the front from the perspective of the other party (user B).

[(11) Monitor function, image correction function]
As shown in FIG. 7, the mobile information terminal 1 of the first embodiment also has a function of monitoring user A's own image. The mobile information terminal 1 displays the monitor image D13 of the user A corresponding to the transmission image D12 in the display screen DP, and asks the user A whether or not the face state of the monitor image D12 is acceptable as the transmission image. confirm. With this monitor function, if user A looks at the monitor image D13 and feels that the distortion-corrected face image of the user A is uncomfortable or dislikes it, the user A can refuse to send the transmission image D12 corresponding to the monitor image D13. can be done. When the mobile information terminal 1 receives the operation of the transmission refusal instruction, it does not transmit the transmission image D12 corresponding to the monitor image D13.

Further, in that case, the mobile information terminal 1 may create a new transmission image D12 based on the registered image D10 instead of the transmission image D12, replace the original transmission image D12, and transmit the new transmission image D12. good. The mobile information terminal 1 may use the face image of the registration image D10 as it is, or may process it to create the transmission image D12. The registered image D10 may be a still image or moving image of the user A's own face captured by an arbitrary camera, or may be any other image (icon image, animation image, etc.) other than the face.

As another function, the mobile information terminal 1 may have an image correction function. The portable information terminal 1 corrects the face image using this function when receiving an instruction to reject transmission of the corrected image D11 (monitor image D13, transmission image D12) once created. At this time, the mobile information terminal 1 applies correction processing to the post-correction image D11 based on the face image of the registered image D10 to create a corrected version of the face image. For example, the portable information terminal 1 processes the state of both eyes in the face image so that the line of sight faces the front, and creates a modified version of the face image. The modified version of the face image may also be displayed on the display screen DP and transmitted to the user A for confirmation.

As a specific example, in a face image of user A captured at a certain timing, when the eye part is not captured well, for example, there is a case where the direction of the line of sight deviates significantly from the front (direction J3). In that case, the mobile information terminal 1 corrects the eye portion of the face image by combining or replacing it with the eye portion of the registered image D10. As a result, the face image after correction is in a state in which the eyes face the front.

Further, when the user A gives an instruction to reject the transmission after confirming the monitor image D13, the mobile information terminal 1 once resets the processing result (the corrected image D11 and the transmission image D12) at that time, and then resets the processing result at a different timing. The generation of the transmission image D12 may be retried based on the input image. The number of retries at that time can also be set. If a sufficient image cannot be finally obtained as a result of retrying up to a predetermined number of times (when user A does not issue a transmission instruction), the mobile information terminal 1 uses the face image of registered image D10 as transmission image D12. good too.

In addition, the mobile information terminal 1 determines whether the image to be the transmission image D12 is an image created in real time or the face image of the registered image D12 according to the user A's touch operation on the area within the display screen DP. may be switched.

[(12) Registered function and registered image]
By using the registration function and the registration image D10, the accuracy of face detection and the like can be improved, and various auxiliary functions can also be used. Data of the registered image D1 is stored in the memory 13, the external memory 16, or the like. A method for registering the face image of the user A as the registered image D10 is as follows. For example, user A operates the registration function of the user setting functions of the videophone application, and uses the normal camera C2 (or the in-camera C3 in Embodiment 2, which will be described later) to image his/her own face from the front. Then, a face image without distortion is registered as a registered image D10. Note that the registration image D10 may be created using the in-camera C1 and the distortion correction function 203, or registration may be performed by reading data from another camera or an external device.

The registered image D10 may include not only a face image captured from the front of the face of the user A, but also a plurality of images captured from various other directions. In this case, the face detection function 201 of the portable information terminal 1 detects the state of the face at that time using the registered image D10 even when the user A changes the direction of the face or moves during the videophone call. can be detected. The image correction function of the mobile information terminal 1 can perform correction processing according to the state of the face.

Further, when the user A confirms the monitor image D13 on the display screen DP and refuses to send it, and uses the registered image D10 instead, the face image selected by the user A from among the plurality of face images of the registered image D10 is used. It can also be a transmission image D12.

In addition, the registered image D10 may include not only a face image of a certain user A, but also a plurality of face images of a plurality of other users who may use the mobile information terminal 1 to make a videophone call. good.

[(13) Personal recognition function]
In Embodiment 1, the face detection function 201 of the photographing processing unit 12 also includes a function (personal recognition function 201B) for recognizing the face of a particular user. The mobile information terminal 1 may not only detect an unspecified face area from a wide-angle image, but also recognize the face of a specific individual user. In that case, the mobile information terminal 1 may detect only the face of a specific user and create the transmission image D12.

A face detection function 201 detects an arbitrary face area from, for example, a wide-angle image. After that, the personal recognition function 201B compares and collates the face area with the face image for personal recognition of the user A registered in advance in the registered image D10. As a result of the comparison, the personal recognition function 201B determines whether the face region in the wide-angle image corresponds to the face of the specific user A based on the similarity. The personal recognition function 201B outputs personal recognition result information.

The portable information terminal 1 applies the control of the videophone function in the hands-free state only when it is the face of the specific user A, and creates, for example, the transmission image D12. When the faces of a plurality of users are shown in the wide-angle image, the transmission image D12 can be created for only the face of a specific user A. For example, the face of a passerby who is only in the background of user A in the wide-angle image need not be treated as an object. As a modification, the individual recognition function 201B may not be provided. Further, the photographing processing unit 12 may perform face recognition of a specific individual on the image before distortion correction, or may perform face recognition of a specific individual on the image after distortion correction. can be

[(14) Face tracking function]
In Embodiment 1, the imaging processing unit 12 (especially the face detection function 201) has a function of automatically tracking the movement of the user's face area based on the moving images (a plurality of images at a predetermined rate) from the in-camera C1. (face tracking function) is also included. By using the wide-angle in-camera C1, the mobile information terminal 1 can track the user by face detection within the wide-angle image, even if the user himself/herself moves a little. The portable information terminal 1 can also make the transmission image D12 tracked such that the user's face is always at the center of the image.

During the videophone call, the user A is not always stationary at the same position, but may be moving. The imaging processing unit 12 detects the face area of the user A for each wide-angle image at each predetermined point in time from the moving image. For example, after the face detection function 201 once detects a face area at a certain time point, the vicinity of the detected face area is searched at a later time point to determine the movement of the face area. As a result, the face area of user A can be continuously tracked on the time axis even when the user is moving while suppressing the amount of image processing.

Also, during the videophone call in the state of FIG. 2, the user A (especially the face) may move from the initial position. For example, user A may temporarily leave the initial position and then return to the initial position. Even in that case, the photographing processing unit 12 tracks the moving face area as much as possible by the face tracking function. When the face of user A is not captured in the wide-angle image, that is, when it cannot be tracked, the photographing processing unit 12 responds, for example, as follows. The photographing processing unit 12 responds using the last detected image in the past and the created transmission image D12. Alternatively, the photographing processing unit 12 responds by temporarily switching to the face image of the registered image D10. When the face of user A is captured again in the wide-angle image, the photographing processing unit 12 detects the face area and tracks it in the same way. Further, even when the face of user A is temporarily hidden by an arbitrary object, the photographing processing unit 11 can deal with the situation by using the face tracking function.

[(15) Other usage states, placement states, and guide functions]
16 to 18 show examples of other usage states and arrangement states of the portable information terminal 1 of the first embodiment. The arrangement state of the terminals when user A makes a videophone call in the hands-free state is not limited to the state shown in FIG. 2, and the following state is also possible.

FIG. 16 shows a first example of states. In FIG. 16 there is an arbitrary object 160, such as a platform, with a slope s5 inclined at some angle 161 with respect to the horizontal plane s0. The user A places the housing of the portable information terminal 1 flat along the slope s5 of the object 160 such as the table. The object 160 and the angle 161 are not particularly limited as long as the housing is stationary. In this state, the user A looks at the display screen DP of the housing in front. The in-camera C<b>1 is arranged in the optical axis direction J<b>2 corresponding to the angle 161 . The in-camera C1 captures the face of the user A (point P1) at the angle of view AV2 in the direction J3. In this way, even if the housing is arranged with a certain degree of inclination, a hands-free videophone can be realized in the same manner as in FIG.

In FIG. 17, as a second example, the position PC1 of the in-camera C1 on the front surface s1 of the housing is positioned in the direction Y2 on the front side in the Y direction as viewed from the user A. FIG. In this case, the face shooting direction J3 (angle θ3) of the in-camera C1 is different from the state shown in FIG. For example, the elevation angle is larger. In this case, in the wide-angle image, the face area of user A is shown inverted. The photographing processing unit 12 can recognize the reversed state from the wide-angle image. The photographing processing unit 12 appropriately performs image reversal processing, and displays the monitor image D13 in the display screen DP with the top and bottom in an appropriate direction.

FIG. 18 shows, as a third example, the case where the longitudinal direction of the housing is arranged along the X direction (horizontal direction as viewed from user A). In this case, the position PC1 of the in-camera C1 on the front surface s1 is positioned on one of the left and right sides (for example, the direction X1) of the user A in the X direction. The photographing processing unit 12 appropriately performs image rotation processing, and displays the monitor image D13 in an appropriate direction within the display screen DP.

In Embodiment 1, the arrangement shown in FIG. 2 is particularly recommended to the user, but the videophone function can be realized in substantially the same manner even in each arrangement described above. Further, in Embodiment 1, regardless of the state of arrangement of the mobile information terminal 1 on the horizontal plane s0 and the positional relationship with the user A in any of the states of FIGS. 2, 17, 18, etc., It can be handled in roughly the same manner. Therefore, user A can freely change the state of the terminal and his/her own position to some extent during the videophone call, which is highly convenient.

Further, the portable information terminal 1 may have a function (guide function) of recommending or guiding the user regarding the layout of the housing. The mobile information terminal 1 uses the camera image and the sensors 30 to grasp the arrangement state of the housing. For example, the mobile information terminal 1 displays guide information (e.g., "It is recommended that the camera be placed on the back side") on the display screen DP so as to recommend the placement state shown in FIG. Audio output may be used.

In addition, when the arrangement state of the housing is not appropriate, the mobile information terminal 1 may output guide information to that effect. For example, when the mobile information terminal 1 is in a certain arrangement state, the angle θ3 with respect to the direction J3 in which the face (point P1) of the user A is photographed from the position PC1 of the in-camera C1 is not within a predetermined angle range (the elevation angle is small). too large or too large), guide information indicating that the placement position is not appropriate may be output.

Further, for example, in the positional relationship of the user A with respect to the mobile information terminal 1, it is assumed that the face cannot be recognized particularly from the in-camera C1. In that case, the mobile information terminal 1 may output guide information to the user A in order to make the positional relationship (terminal position and user position) appropriate. For example, information to the effect that the position or arrangement state of the portable information terminal 1 is to be changed, or information to the effect that the position of the user A's face is to be changed may be output. At this time, if the mobile information terminal 1 knows the positional relationship with the user A, it may output instruction information indicating to which direction or position the mobile information terminal 1 should be changed.

[(16) Comparative example - non-hands-free state]
FIG. 19 shows a normal videophone mode (normal mode) in the mobile information terminal of the comparative example, and a non-handsfree non-handsfree mode as another videophone mode in the first embodiment. In this state or mode, user A holds the casing of the portable information terminal in his/her hand, and both hands are not free. In the example of FIG. 19, the housing is in a vertically upward position. The in-camera CX on the front of the housing is a normal camera having a normal lens with a normal angle of view (so-called narrow angle). This normal angle of view is narrower than the angle of view AV1 of the in-camera C1. The direction JX2 of the optical axis of the in-camera CX is shown, and in this example, it faces the direction Y2 on the front side in the Y direction. The angle of view AVX1 of the in-camera CX is, for example, an angular range from 45 degrees to 135 degrees. Of the angle of view AVX1, the angle of view AVX2 corresponding to the face photographing range is shown.

Also shown are a line-of-sight direction JX1 of the user A, a face photographing direction JX3 of the in-camera CX, and an angle difference AD2 formed therebetween. The greater the angular difference, the more the user A's line of sight is directed downward in the image. When viewed from the other party (user B), the user A's eyes are not facing the front.

On the other hand, in the hands-free mode of FIG. 2 in the first embodiment, the angle difference AD1 can be made smaller than the angle difference AD2 in the comparative example. Therefore, in the hands-free mode of the first embodiment, the direction of the user's line of sight in the image is closer to the front than in the normal mode. As a result, when viewed from the other party (user B), the user A's line of sight is directed more toward the front, so that a more natural and less uncomfortable videophone call is possible.

Note that the in-camera CX of the conventional mobile information terminal usually has a lens, and the shooting range is limited. Therefore, when the user A uses the in-camera CX for the videophone function to make a videophone call in a non-hands-free state while holding the housing with his/her hand, the following considerations and labor are required. User A needs to maintain the positional relationship between the face and the housing while adjusting the orientation of the housing by hand in order to appropriately photograph his own face and convey it to the other party. On the other hand, in the mobile information terminal 1 of Embodiment 1, videophone calls can be made in a hands-free state using the in-camera C1, and the above considerations and troubles are unnecessary.

In the first embodiment, when the normal mode is used, the user A's face is captured at a position near the center of the wide-angle image of the in-camera C1. In the case of this normal mode, the portable information terminal 1 may omit the distortion correction processing and the like, and a different effect can be obtained as an operation different from that in the hands-free mode.

[(17) Effects, etc.]
As described above, according to the mobile information terminal 1 having the videophone function of the first embodiment, a hands-free videophone can be realized with more suitable usability. The user can make videophone calls with both hands free, which is highly convenient. Note that the in-camera C1 on the front surface s1 of the mobile information terminal 1 is a general camera that can be used not only for videophone but also for other purposes (self-portrait, etc.). In Embodiment 1, the in-camera C1 is effectively used to realize a hands-free videophone. The portable information terminal 1 according to the first embodiment does not require deformation of the housing or the use of a separate fixture or the like for hands-free videophone, and is easy to use and highly versatile. .

Further, in the mobile information terminal 1 of Embodiment 1, the optical axis of the camera (in-camera C1) is generally perpendicular to the plane, and the direction of the optical axis of the camera is oblique (for example, 45 degrees). It is also easy to implement, unlike conventional techniques such as those described above and conventional techniques in which the direction of the optical axis of a camera can be changed by mechanical driving.

(Embodiment 2)
A mobile information terminal according to Embodiment 2 of the present invention will be described with reference to FIGS. 20 and 21. FIG. The basic configuration of the second embodiment is the same as that of the first embodiment, and the components of the second embodiment that are different from those of the first embodiment will be described below. The mobile information terminal 1 of Embodiment 2 has a plurality of in-cameras on the front surface s1 of the housing, and uses them properly.

FIG. 20 shows the configuration of the camera section 11 in the portable information terminal 1 of the second embodiment. The camera section 11 includes an in-camera C3 having a normal angle of view in addition to the above-described in-camera C1 and normal camera C2 (which may in particular be a wide-angle camera). The aforementioned in-camera C1 corresponds to the first in-camera, and the in-camera C3 corresponds to the second in-camera.

The shooting processing unit 12 includes an out-camera processing unit 12B that processes normal images from the normal camera C2, a first in-camera processing unit 12A that processes wide-angle images from the in-camera C1, and a first in-camera processing unit 12A that processes normal images from the in-camera C3. It includes a 2-in-camera processing unit 12C and a mode control unit 12D. The photographing processing unit 12 uses the mode control unit 12D to switch which camera among the plurality of cameras is to be used. The mobile information terminal 1 switches the camera mode according to the positional relationship between the terminal and the face of the user A and the positional state of the terminal.

FIG. 21 shows an example of the usage state of the portable information terminal 1 of Embodiment 2, the angle of view of the in-camera C3, and the like. In (A) of FIG. 21, the housing of the mobile information terminal 1 is arranged flat on the horizontal plane s0. For example, on the front surface s1 of the housing, an in-camera C3 (especially a normal lens unit) is provided at a position PC3 near the position PC1 of the in-camera C1. The direction J4 of the optical axis of the in-camera C3 is vertically upward as in the case of the in-camera C1. View angle AV4 (angle range from first angle ANG3 to second angle ANG4) of in-camera C3 is shown. The angle of view AV4 is narrower than the angle of view AV1 of the in-camera C1. For example, the first angle ANG3 is about 60 degrees and the second angle ANG3 is about 135 degrees.

A state 211 is the same as the state of FIG. 2, and shows a case where the face of the user A can be photographed at an angle of view AV2 within the angle of view AV1 of the in-camera C1. A state 212 indicates a positional relationship in which the face of the user A can be photographed by the angle of view AV4 of the in-camera C3. For example, there is a representative point P4 on the face of user A beyond the direction J4 of the optical axis of the in-camera C3.

In the case of a positional relationship in which the face can be photographed with the in-camera C3 as in the state 212, the portable information terminal 1 switches to the camera mode using the in-camera C3 among the plurality of cameras. In addition, as in the state 211, in the positional relationship in which the face can be photographed with an angle of view other than the angle of view AV4 of the angle of view AV1 of the in-camera C1, the portable information terminal 1 switches to the camera mode using the in-camera C1.

FIG. 21B shows an example of the non-hands-free state in the second embodiment, which is close to the non-hands-free state of FIG. For example, when the position of user A's face shifts from state 211 to state 213 in FIG. Control to make a videophone call in free mode (normal mode). Similarly, when the position of user A's face shifts from state 213 to state 211, portable information terminal 1 switches the camera mode from in-camera C3 to in-camera C1 to make a video call in hands-free mode. to control. In accordance with the mode switching, the input image is switched between the wide-angle image captured by the in-camera C1 and the normal image captured by the in-camera C3.

The portable information terminal 1 may automatically select and switch between the two types of camera modes based on the grasp of the state, or may perform the selection and switching based on the instruction operation or setting by the user. . In the mode using the normal image of the in-camera C3, since distortion correction processing or the like is unnecessary, the processing can be made more efficient.

The mobile information terminal 1 uses the face detection function 201 to detect the face area of the user A in the image when the in-camera C1 or the in-camera C3 is in operation, and detects the face area based on the position, direction, and angle of view of the face area. , the camera to be used may be selected and the mode may be switched. For example, the mobile information terminal 1 uses the in-camera C3 depending on whether or not the face is within a predetermined angle of view corresponding to the angle of view AV4 of the in-camera C3, among the angle of view AV1 of the in-camera C1. You may select whether to use the in-camera C1.

As described above, according to the portable information terminal 1 of Embodiment 2, in addition to the same effects as those of Embodiment 1, processing can be made more efficient by using the in-camera C3 with the normal angle of view. Note that the positions of the in-camera C1 and the in-camera C3 on the front surface s1 of the housing of the mobile information terminal 1 are not limited to the configurations described above. It may be the position of the part.

(Other embodiments)
Other embodiments (modifications) of the first and second embodiments are possible as follows.

[Modification (1) - Shooting process]
As shown in FIG. 5 and the like, the photographing processing unit 12 according to the first embodiment detects a face area from a wide-angle image, trims the face area, and then performs distortion correction processing on the trimmed image area. The method of photographing processing is not limited to this and is possible.

FIG. 22 shows an image example of the photographing process in the portable information terminal 1 of the modified example. The photographing processing unit 12 of the mobile information terminal 1 first performs distortion correction processing on the entire wide-angle image G1, and then detects and trims the face area and the like from the image after distortion correction. The portable information terminal 1 applies distortion to a region (range 221) with a horizontal angle of view of 360 degrees or a region with a horizontal angle of view of 180 degrees (range 222 of the lower half of the x-axis) in the wide-angle image G1. Correction processing is performed to obtain a corresponding planarized image GP3. The example of FIG. 22 schematically shows a panorama image in the case of a range 222 with a horizontal angle of view of 180 degrees as the image after distortion correction, which is the flattened image GP3. The mobile information terminal 1 detects an area (eg, area 224) including the face of the user A from the planarized image GP3. Then, the mobile information terminal 1 obtains a trimmed image 226 by trimming the trimming region 225 from the region 224, and creates the transmission image D12 and the like from the trimmed image 226. FIG.

In this modified example, the area of the image region to be subjected to the distortion correction process is larger than that of the first embodiment described above. In the imaging process of Embodiment 1 described above, since the area of the image region to be subjected to the distortion correction process is smaller, it is advantageous in terms of processing efficiency and the like. In the case of terminals with high computational performance, modifications may be adopted. In the modified example, since the target image for face detection is a planarized image, it is advantageous in terms of ease of image processing for face detection.

Further, in another modified example, the photographing processing unit 12 compares and verifies the face image of the registered image D10 of the user A when detecting the face area from the distorted wide-angle image. The facial image of the registered image D10 at that time may be a distorted facial image captured in advance by the in-camera C1.

Further, in another modified example, the mobile information terminal 1 detects a part of the angle of view of the wide-angle image, for example, the range of a semicircle below the x-axis in FIG. 222 (range of elevation angles from 0 degrees to 90 degrees in FIG. 2) may be referenced, the image area to be processed may be narrowed down to that range, and the upper half of the image area may be ignored. Furthermore, as in the example of the area 223, the horizontal angle of view may be narrowed down to a narrower range. Further, for example, when the portable information terminal 1 grasps the state as shown in FIG. 2, the processing target image range may be narrowed down as described above. In the first state of FIG. 2, in the wide-angle image, the face of user A appears in the lower semicircular range 222, and there is almost no possibility of it appearing in the upper semicircular range. Therefore, the above processing is effective.

As another photographing processing method, the portable information terminal 1 uses the distortion correction function 203 to first perform a simple first distortion correction process on the wide-angle image, then perform face detection and trimming, and finally perform trimming. The image may be subjected to a second distortion correction process with higher accuracy.

As a modification, whether or not the image after distortion correction is acceptable as the transmission image D12 may be determined automatically by the mobile information terminal 1 instead of by the user A's confirmation or operation. For example, the portable information terminal 1 compares the face area of the image after distortion correction with the face area of the registered image D10, evaluates the degree of reproduction of the face, and calculates an evaluation value. The mobile information terminal 1 determines that transmission is permitted when the evaluation value is equal to or greater than the set threshold.

[Modification (2) - Object Recognition Function]
In the portable information terminal 1 of the modified example, the photographing processing section 12 (especially the face detection function 201) may have an object recognition function. This object recognition function is a function of recognizing a predetermined object other than a face from a wide-angle image and detecting its object area based on image processing. User A can freely move his or her hand during videophone calls in the hands-free state of FIG. As a result, in the transmission image D12, not only the face of the user A but also any object held in the hand can be shown in the surroundings thereof and can be shown to the other party (user B).

A predetermined object is an object defined on information processing in advance. The imaging processing unit 12 has a detection algorithm corresponding to the object. Predetermined objects include, for example, user's materials, photographs, notebooks, notebook PC screens, user's goods, animals, and the like. A predetermined object is defined as an area having a predetermined shape, such as a rectangle or a circle, and a predetermined color. Based on the detected face area, the photographing processing unit 12 (object recognition function) may search within a predetermined distance range around the face area, for example, to detect the area of the predetermined object.

FIG. 23A shows an image example when using the object recognition function. This image shows the state of a flattened image without distortion. User A talks to the other party (user B) while showing the object 230 during a hands-free videophone call. The object 230 is, for example, an A4-sized document, and has a roughly rectangular shape in the image after distortion correction. The mobile information terminal 1 uses the object recognition function to detect not only the face area 231 but also the area 232 of the specific object 230 from the image. The portable information terminal 1 also performs distortion correction processing and the like on the region 232 of the object 230 . For example, the mobile information terminal 1 may search for a specific object within a range from the point P1 of the face area 231 to a predetermined distance 233 around it. The mobile information terminal 1 may, for example, take a rectangular area 234 that includes the face area 231 and the area 232 of the object 230 to create the transmission image D12. Further, the image is not limited to the image centered on the face region 231 (point P1), and may be a minimum rectangular image that includes the face and the object, such as the region 235 .

Alternatively, the mobile information terminal 1 may divide the face area 231 and the area 232 of the object 230, create respective transmission images D12, display them as monitor images D13 (images 236 and 237), and confirm the transmission. good. Moreover, the mobile information terminal 1 may create an image focused on the detected object 230 (an image enlarged centering on the object 230) as the monitor image D13. Also, by using the aforementioned area A3 or area B3 to secure an area around the face with a large distance, even if the object recognition processing is omitted, the object can be automatically captured within that area.

Since this object recognition function uses a wide-angle image, even if the distance between the face and the object is to some extent, both images can be obtained. For example, FIG. 23B shows a flattened panorama image GP4 based on the wide-angle image. In this panorama image GP4, the face of user A is shown at a position corresponding to directions Y2 and J3 in FIG. 2 and the lower side of the y-axis in FIG. There is a region r1. Then, there is a region r2 in which a predetermined object is captured at a position some distance in the horizontal direction from that position, for example, at a position corresponding to the right side of the x-axis at a horizontal angle of view of 90 degrees. Both of these can be captured in one wide-angle image, and can be the transmission image D12.

As another modification, the photographing processing unit 12 (object recognition function) may detect user A's hands from a wide-angle image, select an area including user A's face and hands, and create transmission image D12. good. Also, when both hands are shown in the wide-angle image, it can be determined that the user is not holding the housing. Therefore, the mobile information terminal 1 may switch to the hands-free mode when both hands of the user A are detected from the wide-angle image.

[Modification (3)-face images of multiple users]
As a modification, for example, one mobile information terminal 1 is placed on a desk, and a plurality of users use the same mobile information terminal 1 to communicate with the other party (user B) as a caller on one transmission side. A method of use for making videophone calls is also possible. In that case, the portable information terminal 1 of the modified example simultaneously performs processing such as face detection and distortion correction on a plurality of faces of a plurality of users within the wide-angle image. Further, at that time, the portable information terminal 1 may separate each user's face in the wide-angle image to create a plurality of transmission images D12, or may create a plurality of transmission images D12 including a plurality of faces. D12 may be created.

(A) of FIG. 24 shows an example of an image including faces of a plurality of users in this modified example. In addition to the main user A, another user C is present as one of the parties making a videophone call. The mobile information terminal 1 detects the user A's face region RU1 and the user C's face region RU2 from within the wide-angle image of the in-camera C1. The mobile information terminal 1, for example, selects an area 241 (for example, a horizontally long rectangle) that includes those two face areas, and creates a transmission image D12. Alternatively, the portable information terminal 1 may take the two face regions as trimming images 242 and 243, create transmission images D12, and display them side by side. Basically the same realization is possible even when there are three or more users, but the number of faces is limited to a predetermined number (for example, four) so that the number of users does not become too large. The mobile information terminal 1 may display the monitor image D13 of each face of a plurality of users in the display screen DP.

In addition, the mobile information terminal 1 performs image processing of the wide-angle image, particularly detection of the state of the mouth, to determine which face of the user is currently speaking among a plurality of faces in the wide-angle image, and determines whether the currently speaking user is speaking. A transmission image D12 or the like of the face of the person may be created. Furthermore, the mobile information terminal 1 may link image processing and voice processing of the microphone 17 to grasp which face of the user is currently speaking among a plurality of faces in the wide-angle image. The portable information terminal 1 may display the monitor images D13 of a plurality of users in parallel on the display screen DP, or may switch and display them on the time axis.

Further, when the faces of the plurality of users are handled, the above processing may be performed only for the plurality of users whose face images are registered in advance as the registered images D10 in the portable information terminal 1. FIG. The portable information terminal 1 does not handle the faces of people (passers-by, etc.) who are not registered in advance. In addition, the portable information terminal 1 may use the registered image D10 as a substitute when the face processing of some users cannot be completed in time, or may use another icon, landscape image, or the like as a substitute. good.

In the image example of (A) above, a plurality of people (user A, user C) are displayed in an area within the angle of view in a certain direction when viewed from the in-camera C1 (for example, the area L1 below the y-axis in (B)). A face is shown. Not limited to this, by using the wide angle of view of the in-camera C1, even if each person is at each position with a different horizontal angle of view around the position of the mobile information terminal 1 on the horizontal plane s0, it can be handled. It is possible. For example, it is possible to deal with any of the areas L1 to L4 above and below the y-axis and on the left and right of the x-axis in the area near the periphery of the wide-angle image G1 in FIG. 24B. That is, a plurality of faces of the plurality of people can be captured in one wide-angle image, and can be used as a transmission image D12.

[Modification (4) - Opponent image correction function]
FIG. 25 shows the partner image correction function provided in the portable information terminal 1 of the modification. When the mobile information terminal 1 of the modified example displays the other party's image in the display screen DP (region R1) as shown in FIG.

FIG. 25A shows an example of a normal partner image received from the mobile information terminal 2 of the partner (user B). In a region R1 within the display screen DP of the mobile information terminal 1, an image g1 of a rectangular rectangle of the other party is displayed.

(B) of FIG. 25 schematically shows how the image g1 of (A), which is diagonally below the user A's eye (point P1), is viewed in the state shown in FIG. In the state of (B), the image g1 appears as a trapezoid whose upper side is smaller than its lower side. That is, when viewed from user A, the side of user B's head appears relatively smaller.

Based on the analysis of the wide-angle image of the in-camera C1 and the information detected by the sensors 30, the mobile information terminal 1 grasps the positional relationship between the user A and the terminal and the arrangement state of the terminal. For example, the portable information terminal 1 estimates the position of the user A's face, the distance from the terminal, and other conditions from the position of the eyes of the user A in the image, the direction of the line of sight, the size of the face, and the like. The mobile information terminal 1 sets the ratio (the ratio between the upper side and the lower side) and the like for inverse keystone correction according to the grasped state. Portable information terminal 1 performs reverse trapezoidal correction processing on the rectangular image received from portable information terminal 2 of the other party in accordance with the ratio and the like to obtain a reverse trapezoidal image. The above ratio may be a preset value.

(C) of FIG. 25 shows an image g1b after inverse keystone correction of the image g1 of (A). This image g1b has an inverted trapezoidal shape, and is a trapezoid with a large upper side and a small lower side. The portable information terminal 1 displays the inverted trapezoidal image g1b in the area R1 of the display screen DP. User A, in the state shown in FIG. 2, sees the other party's image g1b in area R1 in display screen DP obliquely below the eye (point P1). Then, in this state, the other party's image will be seen by the user A in a shape close to a right-angled square as shown in (A). This makes it easier for the user A to visually recognize the other party's image, and is more convenient to use.

[Modification (5)-Three-dimensional image processing function]
As a modification, the mobile information terminal 1 may use a function (three-dimensional image processing function) for processing not only two-dimensional images but also three-dimensional images. For example, the camera section 11 (for example, the normal camera C2) may be equipped with a known infrared camera function and a three-dimensional sensor module. Using this, the imaging processing unit 12 processes the wide-angle image of the in-camera C1 as a three-dimensional image. For example, the portable information terminal 1 irradiates the user's face with, for example, tens of thousands of infrared dots or more by using the infrared camera function and the three-dimensional sensor modules. The portable information terminal 1 takes an image of the infrared dot with an infrared camera, reads subtle irregularities on the face surface from the image, and creates a three-dimensional face map (corresponding three-dimensional image). The mobile information terminal 1 may perform distortion correction processing and the like on the three-dimensional image. Moreover, the mobile information terminal 1 may perform a three-dimensional face correction process by matching the three-dimensional image with the three-dimensional face image information in the registered image D10. In that case, clearer and finer correction can be performed.

Moreover, when performing such three-dimensional advanced correction, the portable information terminal 1 may add analysis using machine learning such as deep learning instead of simply comparing images. For example, the mobile information terminal 1 may incorporate an AI engine (software and hardware that performs deep learning using a convolutional neural network) with a deep learning function. The mobile information terminal 1 uses the AI engine to learn about the user's face from the camera image, thereby improving the performance of face detection and face correction. As a result, specifically, for example, the face of user A is detected, recognized, and corrected in consideration of differences and effects such as changes due to hairstyle and make-up, the presence or absence of glasses and sunglasses, and the degree of growth of the beard. be able to.

Further, the personal identification function 201B of the mobile information terminal 1 may compare and verify the three-dimensional face image of the registered image D10 with the three-dimensional face image photographed by the in-camera C1 and corrected for distortion. As a result, individual recognition as to whether or not user A is the real person can be realized with higher accuracy.

[Modification (6) - directional microphone, directional speaker]
As a modification, the microphone 17 of the mobile information terminal 1 of FIG. 3 may be a directional microphone. Directional microphones include audio processing features such as noise cancellation features. The controller 10 uses the microphone 17 to preferentially collect sound from the direction in which the user A's face is located. The controller 10 cancels noise in the input voice using a noise canceling function to obtain clear voice of the user A. FIG. The controller 10 transmits the voice data to the mobile information terminal 2 of the other party together with the transmission image D12. The direction in which the user A's face is located when viewed from the portable information terminal 1 can be grasped by grasping the state of the portable information terminal 1 or by detecting the face in the image.

As for the microphone 17, a known MEMS microphone or the like may be applied to achieve directivity and noise canceling functions by a known beam forming technique. For example, to achieve a noise canceling function, basically multiple microphones are required. However, if the portable information terminal 1 is small, it may not be possible to mount a plurality of microphones. In that case, the mobile information terminal 1 can separate and emphasize a specific sound source from a plurality of sound sources by using a beam forming technique by installing a MEMS microphone. As a result, it is possible to obtain only user A's voice in an emphasized manner.

Moreover, the mobile information terminal 1 can recognize the position and direction of the user A with a certain degree of accuracy by using the in-camera C1. Therefore, the portable information terminal 1 roughly identifies the position and direction of the user A using the in-camera C1. The mobile information terminal 1 may preferentially emphasize and acquire the voice from the specified position and direction using the microphone 17 and beamforming technology.

Moreover, as a modification, the mobile information terminal 1 may roughly estimate the position and direction of the user A's face based on the analysis of the voice of the microphone 17 . The mobile information terminal 1 may perform processing such as face detection on the wide-angle image according to the position and direction of the face.

Similarly, the speaker 18 may be a directional speaker. The directivity, volume, etc. of the audio output of the speaker 18 may be controlled according to the position of the user A's face relative to the terminal.

Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention.

1... portable information terminal, 2... portable information terminal, s0... horizontal plane, s1... front surface, s2... rear surface, DP... display screen, C1... front camera, J1, J2, J3... direction, AV1, AV2... angle of view, θ1 , θ2... Angle, P1, PC1, PD... Point, ANG1, ANG2... Angle, AD1... Angle difference.

Claims (15)

A mobile information terminal having a videophone function,
Equipped with a first camera at a predetermined position on the front surface of the flat plate housing having a display screen,
When the first user uses the videophone function to make a videophone call with a second user, the housing is placed flat on a first surface of an object and the first camera of the first camera is placed flat on the first surface of the object. A state in which the first user's face is included within the range of the angle of view is defined as a first state,
detecting a first region including the first user's face from the first state image captured by the first camera in the first state, trimming the first image corresponding to the first region, and creating a transmission image to be transmitted to a partner terminal, which is a portable information terminal of the second user, based on the first image, and transmitting the transmission image to the partner terminal;
registering in advance an image containing the face of the first user or an arbitrary image representing the first user as a registered image;
If the first region including the face of the first user cannot be detected from the first state image, the registered image is used to transmit an image that is a substitute for the transmission image.
Personal digital assistant. In the portable information terminal according to claim 1,
performing distortion correction processing on at least the first image of the first state image so as to eliminate or reduce distortion due to the lens of the first camera, and creating the transmission image based on the corrected image;
Personal digital assistant. In the portable information terminal according to claim 2,
detecting the first region including the face of the first user from the first state image, trimming the first image corresponding to the first region, and performing the distortion correction processing on the first image. ,
Personal digital assistant. In the portable information terminal according to claim 2,
performing the distortion correction processing on the first state image, detecting the first area including the face of the first user from the image after distortion correction, and trimming the first image corresponding to the first area. ,
Personal digital assistant. In the portable information terminal according to claim 1,
The first angle of view of the first camera has an angle range from a first angle of 30 degrees or less to a second angle of 150 degrees or more with respect to the first surface,
In the first state, the first angle of view includes a second angle of view that includes the first user's face within an angle range from the first angle to 90 degrees.
Personal digital assistant. In the portable information terminal according to claim 1,
creating a monitor image having image content corresponding to the transmitted image;
displaying the monitor image in an area within the display screen;
determining whether to permit or reject transmission of the transmission image based on confirmation of the monitor image and a predetermined operation by the first user;
Personal digital assistant. In the portable information terminal according to claim 6,
registering in advance an image containing the face of the first user or an arbitrary image representing the first user as a registered image;
In the case of refusal of transmission of the transmission image, transmitting an image as a substitute for the transmission image using the registered image.
Personal digital assistant. In the portable information terminal according to claim 2,
performing trapezoidal correction processing for correcting trapezoidal image content resulting from the positional relationship in the first state on the image after the distortion correction processing;
Personal digital assistant. In the portable information terminal according to claim 1,
When the state of the housing is grasped using a sensor including an acceleration sensor and a gyro sensor, and when it is grasped that it is in the first state, or when a predetermined operation by the first user is received, a non-hands-free operation is performed. Switching from the first mode to a hands-free second mode to create the transmission image;
Personal digital assistant. In the portable information terminal according to claim 1,
creating the transmission image by switching from a non-hands-free first mode to a hands-free second mode according to the position of the first region including the face of the first user in the first state image; conduct,
Personal digital assistant. In the portable information terminal according to claim 1,
Registering in advance an image including the first user's face as a registered image,
When it is recognized that the first user is a specific individual based on the comparison between the first region detected from the first state image and the registered image, the transmission image is created, and the If the first user cannot be recognized as a specific individual, the transmission image is not created;
Personal digital assistant. In the portable information terminal according to claim 1,
detecting a second region including an object held by the first user from the first state image, trimming the second image corresponding to the second region, and generating the transmission image based on the second image; create,
Personal digital assistant. In the portable information terminal according to claim 1,
detecting a plurality of faces of a plurality of persons as the first user from the first state image, and creating the transmission image including the plurality of faces; create a send image,
Personal digital assistant. In the portable information terminal according to claim 1,
When displaying the other party's image received from the other party's terminal in the area within the display screen, displaying the other party's image after inverse keystone correction is performed.
Personal digital assistant. In the portable information terminal according to claim 1,
Based on the first state image or sensor, a positional relationship including the direction of the face of the first user with respect to the housing is estimated, and depending on the estimated direction, the direction of voice input from a microphone or the voice of a speaker. to control the direction of the output,
Personal digital assistant.

Images