JP2014010227A - Portable electronic apparatus, control method therefor and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent visibility of a display screen from being lowered due to shaking during movement.SOLUTION: A portable electronic apparatus includes: a display unit; an imaging unit which is arranged so as to be directed in the same direction as the display unit; a face area detection unit which detects a face area of a user from an image imaged by the imaging unit; a storage unit which stores the face area used as a reference, as a reference face area; a face area analysis unit which analyzes relative relation of the face area with the reference face area; and a display mode change unit which changes a display mode of information to be displayed in the display unit on the basis of the relative relation.

Description

  The present invention relates to a portable electronic device, a control method thereof, and a program. In particular, the present invention relates to a portable electronic device including a display unit, a control method thereof, and a program.

  In recent years, portable electronic devices such as smartphones have been widely used, and users are using portable electronic devices in various situations. Accordingly, there are various distances, angles, and the like at which the user views the display screen of the portable electronic device.

  Patent Document 1 discloses a portable information display terminal that switches a screen display direction based on the orientation of a photographed user's face and the detection result of an inclination sensor. Patent Document 2 discloses a technique for changing the display magnification of the software keyboard screen based on the movement amount of the mobile terminal device.

JP 2011-221094 A JP 2010-282458 A

  The disclosure of the above prior art document is incorporated herein by reference. The following analysis has been made from the viewpoint of the present invention.

  As described above, there are various situations in which a user uses a portable electronic device. For example, the user may use a portable electronic device while walking. Alternatively, the user may use a portable electronic device on a train or the like. In that case, the display screen of the mobile electronic device may shake due to shaking (hand shake) during movement, and the visibility of the display screen may be reduced. For this reason, the user may follow the line of sight (face position and direction) so as to follow the shaking of the display screen. Or a user may fix a portable electronic device to a user's own arm etc., and may prevent a display screen from shaking. For this reason, using a portable electronic device while moving is likely to cause user fatigue.

  Here, with the techniques disclosed in Patent Documents 1 and 2, the orientation and size of the image displayed on the display screen cannot be changed following the shaking during movement. For this reason, the techniques disclosed in Patent Documents 1 and 2 cannot prevent the visibility of the display screen from being lowered due to shaking during movement.

  Accordingly, there is a demand for a portable electronic device, a control method thereof, and a program that prevent the visibility of the display screen from being reduced due to shaking during movement.

  According to the first aspect of the present invention, a face area for detecting a user's face area from a display section, an imaging section arranged to face the same direction as the display section, and an image taken by the imaging section. Based on the relative relationship, a detection unit, a storage unit that stores the reference face region as a reference face region, a face region analysis unit that analyzes a relative relationship of the face region with respect to the reference face region, A portable electronic device is provided that includes a display mode changing unit that changes a display mode of information displayed on the display unit.

  According to a second aspect of the present invention, there is provided a method for controlling a portable electronic device including a display unit and an imaging unit arranged to face the same direction as the display unit. A face area detection step for detecting a face area; a storage step for storing the reference face area as a reference face area; a face area analysis step for analyzing a relative relationship of the face area with respect to the reference face area; There is provided a method for controlling a portable electronic device, including a display mode changing step of changing a display mode of information to be displayed on the display unit based on a relative relationship.

  According to a third aspect of the present invention, there is provided a program that is executed by a computer that controls a portable electronic device that includes a display unit and an imaging unit arranged to face the same direction as the display unit. A face area detection process for detecting a user's face area from an image, a storage process for storing the reference face area as a reference face area, and a face area for analyzing a relative relationship of the face area with respect to the reference face area There is provided a program for executing an analysis process and a display mode changing process for changing a display mode of information to be displayed on the display unit based on the relative relationship. This program can be recorded on a computer-readable storage medium. The storage medium can be non-transient such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, or the like. The present invention can also be embodied as a computer program product.

  According to each aspect of the present invention, a portable electronic device, a control method thereof, and a program for preventing the visibility of a display screen from being lowered due to shaking during movement are provided.

It is a figure for demonstrating one Embodiment. 1 is a plan view illustrating an example of the overall configuration of a mobile electronic device 1. 1 is a diagram illustrating an example of an internal configuration of a mobile electronic device 1. FIG. 3 is a diagram illustrating an example of a usage mode of the mobile electronic device 1. It is a figure which shows an example of a face area | region. It is a figure which shows an example of a feature point and a face area | region. It is a figure which shows an example of the relationship between an angle variation | change_quantity and the angle of a display image. It is a figure which shows an example of a face area | region and a display image. It is a figure which shows an example of a face area | region and a display image. 4 is a flowchart illustrating an example of the operation of the mobile electronic device 1. 4 is a flowchart illustrating an example of the operation of the mobile electronic device 1. It is a figure which shows an example of a face area | region and a display image. It is a figure which shows an example of the correspondence of the angle of a user's face with respect to the display part, and a display mode. It is a figure which shows an example of angle corresponding | compatible information.

  First, an outline of an embodiment will be described with reference to FIG. Note that the reference numerals of the drawings attached to the outline are attached to the respective elements for convenience as an example for facilitating understanding, and the description of the outline is not intended to be any limitation.

  As described above, there are various situations in which a user uses a portable electronic device, and the portable electronic device may be used while moving. However, when a portable electronic device is used during movement, the distance and angle between the user and the display screen frequently change due to camera shake or the like. As a result, when the mobile electronic device is used during movement, the visibility of the display screen may be reduced. Therefore, there is a demand for a portable electronic device that prevents the visibility of the display screen from being lowered due to shaking during movement.

  Therefore, as an example, the portable electronic device 100 shown in FIG. 1 is provided.

  The mobile electronic device 100 includes a display unit 101, an imaging unit 102 arranged to face the same direction as the display unit 101, and a face region detection unit 103 that detects a user's facial region from an image captured by the imaging unit 102. And a storage unit 104 that stores the reference face region as a reference face region, a face region analysis unit 105 that analyzes the relative relationship of the face region with respect to the reference face region, and a display on the display unit 101 based on the relative relationship A display mode changing unit 106 that changes a display mode of information to be displayed.

  The mobile electronic device 100 analyzes the photographed user's face area, and changes the display mode of information displayed on the display unit 101 based on the analysis result. Specifically, first, the face area detection unit 103 detects the user's face area from the image captured by the imaging unit 102. Here, the size, angle, and the like of the detected face area change according to the movement of the portable electronic device 100 or the user's face. Therefore, the face area analysis unit 105 analyzes the relative relationship between the reference face area stored in advance in the storage unit 104 and the detected face area. And the portable electronic device 100 changes the display mode of the information displayed on the display part 101 based on the analysis result of the face area | region analysis part 105 so that visibility may become high. Therefore, the portable electronic device 100 can prevent the visibility of the display unit 101 from being reduced due to shaking during movement.

  Hereinafter, specific embodiments will be described in more detail with reference to the drawings.

[First Embodiment]
The first embodiment will be described in more detail with reference to the drawings.

  FIG. 2 is a plan view showing an example of the overall configuration of the portable electronic device 1 according to the present embodiment.

  The portable electronic device 1 includes an imaging unit 10, a display unit 20, and an operation unit 30. The portable electronic device 1 according to the present embodiment is not intended to be limited to the form shown in FIG. Examples of the mobile electronic device 1 include a smartphone, a mobile phone, a game machine, a tablet PC (Personal Computer), and a notebook PC.

  The imaging unit 10 is arranged so as to face the same direction as the display unit 20, and can capture the face of the user facing the display unit 20. Hereinafter, an image captured by the imaging unit 10 is referred to as a captured image. The imaging unit 10 is preferably arranged on the same plane as the display unit 20.

  The user visually recognizes information (such as characters and photographs) displayed on the mobile electronic device 1 through the display unit 20. As the display unit 20, a liquid crystal panel, an organic EL (Electro Luminescence) panel, or the like can be used.

  The operation unit 30 receives a user operation on the mobile electronic device 1. In FIG. 2, hardware keys are illustrated as the operation unit 30, but operation means such as a touch panel may be employed.

  FIG. 3 is a diagram illustrating an example of the internal configuration of the mobile electronic device 1. The portable electronic device 1 includes an imaging unit 10, a display unit 20, an operation unit 30, a control unit 40, and a storage unit 50. In FIG. 3, for the sake of simplicity, modules related to the portable electronic device 1 according to the present embodiment are mainly described. In addition, since the display part 20 and the operation part 30 are as above-mentioned, further description is abbreviate | omitted.

  The imaging unit 10 captures the user's face as described above. The captured image is stored in the storage unit 50. The captured image stored in the storage unit 50 can be accessed from the control unit 40. Note that the imaging unit 10 may be activated based on a user operation.

  FIG. 4 is a diagram illustrating an example of how the mobile electronic device 1 is used by the user. FIG. 4A is a diagram illustrating a case where the user holds the portable electronic device 1 from the side. FIG. 4B is a diagram illustrating an example of the captured image 201 in the case of FIG. FIG. 4C is a diagram illustrating an example of an image displayed on the display unit 20 in the state of FIG. In the following description, an image displayed on the display unit 20 is referred to as a display image.

  In the following description, as shown in FIG. 4B, the upper left vertex of the photographed image 201 is set to the origin (X, Y) = (0, 0). From the origin, the horizontal direction is the X axis and the vertical direction is the Y axis. Further, as shown in FIG. 4C, the upper left vertex of the display unit 20 is set to the origin (x, y) = (0, 0). The horizontal direction from the origin is the x axis, and the vertical direction is the y axis. Note that, as shown in FIG. 4B, the right direction of the display unit 20 is positive in the X axis. In the Y axis, the downward direction of the display unit 20 is positive. Further, as shown in FIG. 4C, the x axis is positive in the right direction of the display unit 20. In the y-axis, the downward direction of the display unit 20 is positive.

  The control unit 40 controls not only the functional module such as the imaging unit 10 illustrated in FIG. 2 but also the entire portable electronic device 1. The control unit 40 includes a feature point detection unit 41, a face region detection unit 42, a face region analysis unit 43, a display mode change unit 44, and an image signal generation unit 45.

  The feature point detection unit 41 extracts a plurality of feature points having predetermined features from the captured image. Specifically, the feature point detection unit 41 extracts an eye or the like as a feature point from the captured image.

  The face area detection unit 42 detects the user's face area from the captured image. Specifically, the face area detection unit 42 detects an area including the feature point detected by the feature point detection unit 41 as a face area. For example, the face region is preferably a region including eye and mouth feature points.

  FIG. 5 is a diagram illustrating an example of a face area detected by the face area detection unit 42. In FIG. 5A, the face area 301 is an area including the eyes and mouth in the captured image 201. On the other hand, in FIG. 5B, the face area 302 is an area including the eyes in the captured image 201.

  The face area analysis unit 43 analyzes the face area. Specifically, the face area analysis unit 43 acquires information including the size and angle of the face area. Hereinafter, the face area to be analyzed by the face area analysis unit 43 is referred to as a target face area. Details of the face area analysis unit 43 will be described later.

  The display mode changing unit 44 determines the display mode of information (display image) provided to the user based on the analysis result of the face area analyzing unit 43. Specifically, the display mode changing unit 44 changes the angle and size of the display image.

  The image signal generation unit 45 generates an image signal of a display image based on the determined display mode.

  The storage unit 50 stores a captured image captured in advance at a predetermined position and angle as a reference captured image. Specifically, first, the user holds the portable electronic device 1 at a position and an angle at which the display unit 20 has high visibility. Then, the imaging unit 10 captures the user's face based on an operation such as pressing the user's operation key (operation unit 30). Hereinafter, the face area detected from the reference photographed image is referred to as a reference face area. Further, hereinafter, the operation for determining the reference face area is referred to as a reference face area determining operation. As shown in FIG. 4A, the reference photographed image is preferably an image photographed by the user facing the portable electronic device 1.

  Note that when the reference captured image is captured, the distance between the user's face and the display unit 20 is different for each user. For example, a myopic user often holds the portable electronic device 1 so that the distance between the display unit 20 and the user's face is short. On the other hand, a hyperopic user often holds the portable electronic device 1 so that the distance between the display unit 20 and the user's face is increased.

  Here, the mobile electronic device 1 may include a reference image shooting mode, for example. Specifically, in the reference image shooting mode, the control unit 10 displays sample characters, images, and the like on the display unit 20. Then, the user holds the portable electronic device 1 at a position and an angle at which it is easy to visually recognize characters and images displayed on the display unit 20. And the imaging part 10 image | photographs a user's face based on a user's operation.

  Next, the change in the display mode of the display image will be described in detail.

  As described above, the display mode changing unit 44 determines the display mode of the display image based on the analysis result of the face area analyzing unit 43. Here, the face area analysis unit 43 analyzes the relative relationship of the target face area with respect to the reference face area.

  First, the angle of the display image will be described.

  The face area analysis unit 43 calculates the angle (inclination) of the target face area with respect to the reference face area. Hereinafter, the angle of the target face area with respect to the reference face area is referred to as an angle change amount. Then, the display mode changing unit 44 rotates the display image according to the angle of the target face area.

  Specifically, the face area analysis unit 43 extracts one axis (hereinafter referred to as a first reference axis) from the reference face area. The first reference axis may be, for example, a line segment having two feature points in the reference face area as end points. Alternatively, the first reference axis may be a line segment that connects the end points of the left and right eyes. Alternatively, the first reference axis may be a line segment connecting feature points such as the nose and mouth.

  Further, the face area analysis unit 43 extracts one axis (hereinafter referred to as a second reference axis) from the target face area. Here, the face area analysis unit 43 needs to extract the second reference axis under the same conditions as the first reference axis. For example, when the first reference axis is a line segment connecting the left and right eye end points, the second reference axis needs to be a line segment connecting the left and right eye end points.

  Then, the face area analysis unit 43 calculates an angle formed by the second reference axis with respect to the first reference axis as an angle change amount.

  FIG. 6 is a diagram illustrating an example of feature points detected by the feature point detection unit 41 and a face area. FIG. 6A is a diagram illustrating an example of the reference face area 311. Point P401 and point P402 shown in FIG. 6A are feature points representing the nose and mouth. On the other hand, FIG. 6B is a diagram illustrating an example of the target face area 312. In FIG. 6B, the target face area 312 rotated with respect to the orientation of the reference face area 311 is shown. A point P411 and a point P412 shown in FIG. 6B are feature points corresponding to the point P401 and the point P402.

  Next, the relationship between the angle change amount and the angle of the display image will be described with reference to FIG. FIG. 7A is a diagram showing an example in which the feature points shown in FIG. 6 are projected onto the XY plane. Here, a line segment with the points P401 and P402 as end points is defined as a line segment L1. Furthermore, a line segment having the end points of the points P411 and P412 is defined as a line segment L11. The angle formed between the line segment L1 and the line segment L11 is defined as an angle θ. That is, the angle θ corresponds to the amount of change in angle.

  Then, the display mode changing unit 44 rotates the display image of the reference face area at the rotation angle θ. That is, the display mode changing unit 44 rotates the display image according to the angle of the user's face. For example, as shown in FIG. 7B, the display image 502 in the target face area is an image obtained by rotating the display image 501 in the reference face area at the rotation angle θ.

  FIG. 8 is a diagram illustrating an example of face areas with different inclinations and corresponding display images. FIG. 8A shows an example of the reference face area 321. FIG. 8B is a diagram illustrating an example of a display image in the case of FIG. FIG. 8C is a diagram illustrating an example of the target face area 322. FIG. 8D is a diagram illustrating an example of a display image in the case of FIG. As shown in FIG. 8D, even when the user tilts his / her face, the portable electronic device 1 displays the display image so that the user's face and the display image are parallel to each other. Therefore, the portable electronic device 1 can prevent a decrease in visibility when the user tilts his / her face.

  Next, the size of the display image will be described.

  The face area analysis unit 43 calculates the ratio of the area of the target face area to the reference face area. Then, the display mode changing unit 44 changes the size of the display image in accordance with the area ratio calculated by the face area analyzing unit 43. For example, when the area of the target face area is smaller than the reference face area, the user is considered to be relatively far from the display unit 20. Therefore, it is preferable that the display image is displayed larger than the reference face area. On the other hand, when the area of the target face area is larger than the reference face area, it is considered that the user is relatively close to the display unit 20. Therefore, the display image is preferably displayed smaller than the reference face area.

  Here, the display mode changing unit 44 preferably determines the size of the display image with respect to the ratio of the area of the face area based on a predetermined correspondence relationship. Specifically, the display mode changing unit 44 may determine the size of the display image based on a predetermined linear function indicating the correspondence between the area ratio of the face area and the size of the display image. . Or the memory | storage part 50 may memorize | store the correspondence of the area ratio of a face area | region, and the magnitude | size of a display image previously.

  FIG. 9 is a diagram illustrating an example of face areas having different areas and corresponding display images. FIG. 9A is a diagram illustrating an example of the reference face area 331. FIG. 9B is a diagram illustrating an example of a display image in the case of FIG. FIG. 9C is a diagram illustrating an example of the target face area 332. FIG. 9D is a diagram illustrating an example of a display image in the case of FIG. Here, the area of the target face area 332 is smaller than the area of the reference face area 331. Here, the target face area 332 is an image obtained by reducing the reference face area 331 by r%. Therefore, as shown in FIG. 9D, the display mode changing unit 44 enlarges and displays the display image of FIG. 9C. Specifically, in the case of FIG. 9, the display mode changing unit 44 displays the display image of FIG. Therefore, in the portable electronic device 1, it is possible to prevent the visibility from being lowered due to a change in the distance between the user's face and the display unit 20.

  Next, the operation of the mobile electronic device 1 will be described.

  FIG. 10 is a flowchart illustrating an example of an operation related to a change in display mode.

  In step S01, the imaging unit 10 captures the user's face.

  In step S02, the face area detection unit 42 determines whether or not to detect the target face area. That is, the face area detection unit 42 determines whether or not the feature point detection unit 41 detects a predetermined feature point. If the target face area is detected (Yes branch in step S02), the process proceeds to step S03. If the target face area is not detected (No branch in step S02), the process proceeds to step S05.

  In step S03, the face area analysis unit 43 analyzes the target face area.

  In step S04, the display mode changing unit 44 changes the display mode of the display image based on the analysis result of the face area analyzing unit 43.

  In step S05, the image signal generation unit 45 generates an image signal of the display image based on the determined display mode.

  In step S06, the control unit 40 determines whether or not the operation unit 30 has detected an end operation. For example, the end operation may be an operation in which the user presses an end button or the like when the display mode changing process is stopped. Alternatively, the control unit 40 may change the display mode while a predetermined application (e-mail application, Web application, etc.) is being executed. In this case, the end operation may be an operation that ends the application. When the end operation is detected (Yes branch in step S06), the control unit 40 ends the display mode changing process. When the end operation is not detected (No branch in step S06), the process proceeds to step S01 and the process is continued.

  For example, the mobile electronic device 1 may include a display mode change mode. Specifically, when the mobile electronic device 1 is set in the display mode change mode, the display mode of the display image may be changed based on the processing in steps S01 to S06.

  Next, FIG. 11 is a flowchart illustrating an example of an operation for determining a reference face area based on a user operation.

  In step S101, the imaging unit 10 captures the user's face.

  In step S102, the face area detection unit 42 determines whether or not a face area has been detected. If a face area is detected (Yes branch in step S102), the process proceeds to step S103. When the face area is not detected (No branch in step S102), the process proceeds to step S101 and the process is continued.

  In step S <b> 103, the control unit 40 determines whether or not the operation unit 30 has detected a reference face region determination operation. When the determination operation is detected (Yes branch in step S103), the process proceeds to step S104. When the determination operation is not detected (No branch in step S103), the process proceeds to step S101 and the process is continued.

  In step S104, the control unit 40 stores the face area detected by the face area detection unit 42 in the storage unit 50 as a reference face area.

[Modification 1]
As a first modification of the portable electronic device 1 according to the first embodiment, an image obtained from the imaging unit 10 may be sequentially analyzed, and shooting may be started when a user's face is recognized in the image. Furthermore, when there is a human sensor (infrared sensor or the like) arranged so as to face the same direction as the imaging unit 10, some object is within a predetermined range from the output of such an infrared sensor. Shooting may be started when it is determined that the detection has been made.

  As described above, in the mobile electronic device 1 according to the present embodiment, the display mode of the display image is changed according to the angle of the user's face and the distance from the display unit 20. For example, the relative distance and angle between the display unit 20 and the user's face change due to shaking during movement. As a result, the visibility of the display screen of the portable electronic device may be reduced. Here, in the portable electronic device 1 according to the present embodiment, the size and angle of the display image are changed according to changes in the relative distance and angle between the display unit 20 and the user's face. Therefore, in the portable electronic device 1 according to the present embodiment, it is possible to prevent the visibility of the display screen from being lowered due to shaking during movement.

[Second Embodiment]

  Next, the second embodiment will be described in detail with reference to the drawings.

  In the second embodiment, the position of the display image is changed according to the position of the face area. In the description of the present embodiment, the description of the same parts as those in the first embodiment is omitted. Further, in the description of the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  When the user uses the portable electronic device 1, it is preferable that a display image is displayed in front of the user. Therefore, when the position of the user's face moves up, down, left, and right, the position of the display image is preferably changed according to the position of the user's face. Therefore, in the portable electronic device 1 according to the present embodiment, the display mode changing unit 44 changes the position of the display image according to the position of the target face area.

  The face area analysis unit 43 calculates the amount of displacement of the target face area on the XY plane with respect to the reference face area. The display mode changing unit 44 changes the position of the display image based on the calculated displacement amount. Here, the face area analysis unit 43 needs to normalize the target face area before calculating the displacement amount. That is, the face area analysis unit 43 normalizes the rotation angle and area of the target face area.

  Specifically, first, the face area analysis unit 43 rotates the target face area so that the rotation angle of the target face area matches the rotation angle of the reference face area based on the angle change amount. Furthermore, the face area analysis unit 43 performs an enlargement or reduction process on the target face area so that the areas of the target face area with respect to the reference face area match.

  Next, the calculation of the displacement amount will be described.

  First, the face area analysis unit 43 extracts one point (hereinafter referred to as a first reference point) included in the reference face area. The first reference point may be a centroid point of the reference face area, for example. Alternatively, it may be a feature point detected by the face area detection unit 42.

  Further, the face area analysis unit 43 extracts one point (hereinafter referred to as a second reference point) included in the target face area. The second reference point needs to be extracted under the same conditions as the first reference point. For example, when the first reference point is the centroid point of the reference face area, the second reference point needs to be the centroid point of the target face area.

  Then, the face area analysis unit 43 calculates a displacement amount of the second reference point with respect to the first reference point. The display mode changing unit 44 changes the position of the display image based on the calculated displacement amount.

  The display mode changing unit 44 preferably changes the position of the display image based on at least one of the parallel movement of the image, the scroll, and the shift of the character string.

  FIG. 12 is a diagram illustrating an example of a correspondence relationship between the face area and the position of the display image. FIG. 12A is a diagram illustrating an example of the reference face area 341. FIG. 12B is a diagram illustrating an example of a display image in the case of FIG. FIG. 12C is a diagram illustrating an example of the target face area 342. FIG. 12D is a diagram illustrating an example of a display image in the case of FIG. As shown in FIG. 12B, the target face area 342 exists at the upper right position with respect to the reference face area 341. Accordingly, as shown in FIG. 12D, in the case of the position of the target face area 342, the display image is displayed on the upper right of the display unit 20.

  As described above, in the portable electronic device 1 according to the present embodiment, the position of the display image is changed according to the position of the user's face. Therefore, in the portable electronic device 1 according to the present embodiment, it is possible to further prevent the visibility of the display screen from being lowered due to shaking during movement.

[Third Embodiment]

  Subsequently, the third embodiment will be described in detail with reference to the drawings.

  In the third embodiment, the display mode is changed when the user tilts and holds the portable electronic device 1. In the description of the present embodiment, the description of the same parts as those in the first embodiment is omitted. Further, in the description of the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  FIG. 13 is a diagram illustrating an example of a correspondence relationship between a posture in which the user holds the portable electronic device 1 and a display mode. FIG. 13A is a diagram showing a user who faces the mobile electronic device 1 from the side. FIG. 13B is a diagram illustrating an example of the reference face area 351. FIG. 13C is a diagram illustrating an example of a display image in the case of FIG.

  Furthermore, FIG.13 (d) is a figure which shows the user who inclines and holds the portable electronic device 1 from a side surface. In FIG. 13D, the user holds down the mobile electronic device 1 while looking down at the display unit 20. FIG. 13E is a diagram showing the target face area 352 in the case of FIG. In the case of FIG. 13D, the imaging unit 10 captures the face of the user viewed from below as the captured image 201. FIG. 13F is a diagram illustrating an example of a display image in the case of FIG. For example, in FIG. 13F, a display image is displayed on the upper portion of the display unit 20 as compared with the case of FIG. Thus, in the portable electronic device 1 according to the present embodiment, the display mode is changed according to the angle of the user's face with respect to the display unit 20.

  Specifically, in the portable electronic device 1 according to the present embodiment, a correspondence relationship between a predetermined face area and a display mode (hereinafter referred to as angle correspondence information) is stored in the storage unit 50 in advance. Here, the storage unit 50 preferably stores, as angle correspondence information, a correspondence relationship between a face area of a user having a different orientation and a display mode. In addition, it is preferable that the angle correspondence information includes any information on the rotation angle, size, and position of the display image as a display mode.

  Then, the face area analysis unit 43 collates the face area stored as the angle correspondence information with the target face area detected by the face area detection unit 42. Then, the display mode changing unit 44 selects a corresponding display mode from the angle correspondence information based on the collation result. For face area matching, for example, template matching may be performed using a stored face area as a template. Or you may collate a face area | region based on the positional relationship of feature points, such as eyes and a mouth. There are various face area collation methods, but any method can be used.

  FIG. 14 is a diagram illustrating an example of angle correspondence information. In FIG. 14, the correspondence between the face area image and the display mode is shown as the angle correspondence information. Further, in FIG. 14, “position” indicates the position of the display image on the xy plane. “Angle” indicates an angle change amount. “Enlargement factor” indicates the enlargement factor of the display image. Further, as described above, the x axis is positive in the right direction of the display unit 20. Further, the y-axis is positive in the downward direction of the display unit 20.

  Furthermore, FIG. 14A1 shows the correspondence between the reference face area and the display mode. On the other hand, FIGS. 14 (a2) to (a5) show the correspondence between the user's face area taken from the top, bottom, left and right and the display mode. In the case of FIGS. 14A2 to 14A5, the angle change amount is 0 degree, the enlargement ratio is 120%, and the display image is displayed.

  For example, in the case of FIG. 14A2, it is a face area detected by photographing the user's face from below. In this case, the position of the display image is (x, y) = (50, 20). On the other hand, in the case of FIG. 14A1, the position of the display image is (x, y) = (50, 50). Therefore, in the case of FIG. 14 (a2), it means that the display image is displayed on the upper side as compared with the case of FIG. 14 (a1).

  In the case of FIG. 14 (a3), it means that the display image is displayed on the upper side as compared with the case of FIG. 14 (a1). Further, in the case of FIG. 14 (a4), it means that the display image is displayed on the left side as compared with the case of FIG. 14 (a1). Further, in the case of FIG. 14 (a5), it means that the display image is displayed on the right side as compared with the case of FIG. 14 (a1).

  As described above, in the portable electronic device 1 according to the present embodiment, the display mode is adjusted according to the angle of the user's face even when the user tilts and holds the portable electronic device 1. Therefore, in the portable electronic device 1 according to the present embodiment, it is possible to further prevent a reduction in the visibility of the display screen in response to various changes in the user's posture.

  A part or all of the above embodiments can be described as in the following supplementary notes, but is not limited thereto.

  (Supplementary Note 1) A display unit, an imaging unit arranged to face the same direction as the display unit, a face region detection unit that detects a user's facial region from an image captured by the imaging unit, and a reference A storage unit that stores the face region as a reference face region, a face region analysis unit that analyzes a relative relationship of the face region with respect to the reference face region, and information to be displayed on the display unit based on the relative relationship A portable electronic device comprising: a display mode changing unit that changes a display mode.

  (Supplementary Note 2) A portable electronic device including a feature point detection unit that detects a plurality of feature points from an image captured by the imaging unit, wherein the face region detection unit detects a region including the feature points as the face region. .

  (Additional remark 3) The said face area analysis part calculates the angle which the 2nd reference axis in the said face area | region makes with respect to the 1st reference axis in the said reference face area as an angle variation | change_quantity, The said display The aspect changing unit is a portable electronic device that rotates an image to be displayed on the display unit based on the angle change amount.

  (Additional remark 4) The said face area analysis part calculates the ratio of the area of the said face area with respect to the said reference | standard face area, and the said display mode change part is based on the ratio of the area of the image displayed on the said display part. A portable electronic device that changes size.

  (Additional remark 5) The said face area analysis part calculates the displacement amount of the 2nd reference point contained in the said face area with respect to the 1st reference point contained in the said reference face area, The said display mode change part Is a portable electronic device that changes the position of an image to be displayed on the display unit based on the amount of displacement.

  (Additional remark 6) The said reference face area is a portable electronic device which is the said face area which the said face area detection part detected from the image image | photographed with the predetermined | prescribed position and angle.

  (Additional remark 7) The said memory | storage part memorize | stores the correspondence of the said predetermined face area | region and the said display mode as angle corresponding | compatible information, and the said display mode change part respond | corresponds to the said face area | region from the said angle corresponding | compatible information. A portable electronic device that selects the display mode.

  (Additional remark 8) The said angle corresponding | compatible information is a portable electronic device containing the information of at least any one of the rotation angle of an image, a magnitude | size, and a position as the said display mode.

  (Additional remark 9) It is a control method of a portable electronic device provided with a display part and the imaging part arrange | positioned so that it may face the same direction as the said display part, Comprising: The face area which detects a user's face area from a picked-up image Based on the detection step, the storage step of storing the reference face region as a reference face region, the face region analysis step of analyzing the relative relationship of the face region with respect to the reference face region, A display mode changing step of changing a display mode of information to be displayed on the display unit.

  (Supplementary note 10) A method for controlling a portable electronic device, including a feature point detection step of detecting a plurality of feature points from a photographed image, wherein the face region detection step detects a region including the feature points as the face region.

  (Additional remark 11) The said face area analysis process calculates the angle which the 2nd reference axis in the said face area | region makes with respect to the 1st reference axis in the said reference face area as an angle variation | change_quantity, The said display The aspect changing step is a method of controlling a portable electronic device that rotates an image to be displayed on the display unit based on the angle change amount.

  (Additional remark 12) The said face area analysis process calculates ratio of the area of the said face area with respect to the said reference | standard face area, and the said display mode change process is based on the ratio of said area, and displays the image displayed on the said display part. A method for controlling a portable electronic device whose size is changed.

  (Additional remark 13) The said face area analysis process calculates the displacement amount of the 2nd reference point contained in the said face area with respect to the 1st reference point contained in the said reference face area, The said display mode change process The control method of the portable electronic device which changes the position of the image displayed on the said display part based on the said displacement amount.

  (Additional remark 14) The said memory | storage process memorize | stores the correspondence of the said predetermined face area | region and the said display mode as angle corresponding | compatible information, and the said display mode change process respond | corresponds to the said face area | region from the said angle corresponding | compatible information. A method for controlling a portable electronic device that selects the display mode.

  (Supplementary Note 15) A program that is executed by a computer that controls a portable electronic device that includes a display unit and an imaging unit that is arranged so as to face the same direction as the display unit. A face area detection process for detecting a reference area, a storage process for storing the reference face area as a reference face area, a face area analysis process for analyzing a relative relationship of the face area with respect to the reference face area, and the relative relationship And a display mode changing process for changing a display mode of information to be displayed on the display unit.

  (Additional remark 16) The program which performs the feature point detection process which detects a some feature point from a picked-up image, and the said face area detection process detects the area | region containing the said feature point as said face area.

  (Additional remark 17) The said face area analysis process calculates the angle which the 2nd reference axis in the said face area | region makes with respect to the 1st reference axis in the said reference face area as an angle variation | change_quantity, The said display The aspect change process is a program for rotating an image to be displayed on the display unit based on the angle change amount.

  (Additional remark 18) The said face area analysis process calculates ratio of the area of the said face area with respect to the said reference | standard face area, and the said display mode change process is based on the ratio of the area, and displays the image displayed on the said display part. A program that changes the size.

  (Additional remark 19) The said face area analysis process calculates the displacement amount of the 2nd reference point contained in the said face area with respect to the 1st reference point contained in the said reference face area, The said display mode change process The program which changes the position of the image displayed on the said display part based on the said displacement amount.

  (Additional remark 20) The said memory | storage process memorize | stores the correspondence of the said predetermined face area | region and the said display mode as angle corresponding | compatible information, and the said display mode change process respond | corresponds to the said face area | region from the said angle corresponding | compatible information. A program for selecting the display mode to be performed.

  Each disclosure of the cited patent documents and the like cited above is incorporated herein by reference. Within the scope of the entire disclosure (including claims) of the present invention, the embodiments and examples can be changed and adjusted based on the basic technical concept. Various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) within the scope of the claims of the present invention, Selection is possible. That is, the present invention of course includes various variations and modifications that could be made by those skilled in the art according to the entire disclosure including the claims and the technical idea.

DESCRIPTION OF SYMBOLS 1,100 Portable electronic device 10,102 Image pick-up part 20,101 Display part 30 Operation part 40 Control part 41 Feature point detection part 42,103 Face area detection part 43,105 Face area analysis part 44,106 Display mode change part 45 Image Signal generation unit 50, 104 Storage unit 201 Captured image 301, 302 Face area 301, 311, 321, 331, 341, 351 Reference face area 302, 312, 322, 332, 342, 352 Target face area 501 Reference face area Display image 502 Display image in the target face area

Claims (10)

A display unit;
An imaging unit arranged to face the same direction as the display unit;
A face area detection unit that detects a user's face area from an image captured by the imaging unit;
A storage unit for storing the reference face area as a reference face area;
A face area analysis unit for analyzing a relative relationship of the face area with respect to the reference face area;
A display mode changing unit that changes a display mode of information to be displayed on the display unit based on the relative relationship;
A portable electronic device comprising: A feature point detection unit that detects a plurality of feature points from an image captured by the imaging unit;
The portable electronic device according to claim 1, wherein the face area detection unit detects an area including the feature point as the face area. The face area analysis unit calculates an angle formed by a second reference axis in the face area with respect to a first reference axis in the reference face area as an angle change amount;
The portable electronic device according to claim 1, wherein the display mode changing unit rotates an image displayed on the display unit based on the angle change amount. The face area analysis unit calculates a ratio of the area of the face area to the reference face area;
The portable electronic device according to any one of claims 1 to 3, wherein the display mode changing unit changes a size of an image displayed on the display unit based on the ratio of the areas. The face area analyzing unit calculates a displacement amount of a second reference point included in the face area with respect to a first reference point included in the reference face area;
The portable electronic device according to any one of claims 1 to 4, wherein the display mode changing unit changes a position of an image displayed on the display unit based on the displacement amount.   The portable electronic device according to claim 1, wherein the reference face area is the face area detected by the face area detection unit from an image photographed at a predetermined position and angle. The storage unit stores a correspondence relationship between the predetermined face area and the display mode as angle correspondence information,
The portable electronic device according to any one of claims 1 to 6, wherein the display mode changing unit selects the display mode corresponding to the face area from the angle correspondence information.   The portable electronic device according to claim 7, wherein the angle correspondence information includes at least one of a rotation angle, a size, and a position of an image as the display mode. A control method for a portable electronic device comprising: a display unit; and an imaging unit arranged to face the same direction as the display unit,
A face area detection step of detecting a user's face area from the captured image;
Storing the face area as a reference as a reference face area;
A face area analysis step of analyzing a relative relationship of the face area with respect to the reference face area;
A display mode changing step for changing a display mode of information to be displayed on the display unit based on the relative relationship;
A method for controlling a portable electronic device, comprising: A program to be executed by a computer that controls a portable electronic device comprising: a display unit; and an imaging unit arranged to face the same direction as the display unit,
A face area detection process for detecting a user's face area from a captured image;
A storage process for storing the reference face area as a reference face area;
A face area analysis process for analyzing a relative relationship of the face area with respect to the reference face area;
A display mode changing process for changing a display mode of information to be displayed on the display unit based on the relative relationship;
A program that executes.

Images