WO2013069796A1

WO2013069796A1 - Electronic device, information system and server

Info

Publication number: WO2013069796A1
Application number: PCT/JP2012/079222
Authority: WO
Inventors: 井上　英也
Original assignee: 株式会社ニコン
Priority date: 2011-11-10
Filing date: 2012-11-12
Publication date: 2013-05-16
Also published as: JP6146307B2; JP2019117378A; JPWO2013069796A1; JP6828758B2; JP2017207753A; JP6477782B2

Abstract

An electronic device is provided with: a display unit for displaying a display image based on display data; an image capture unit which can capture an image of a subject in front of the display unit; and a display control unit for controlling the display format of the display image in accordance with the amount of change over time of a feature portion of the subject which has been captured in front.

Description

Electronic device, information system and server

The present invention relates to an electronic device, an information system, and a server.
This application claims priority based on Japanese Patent Application No. 2011-246571 filed on November 10, 2011, the contents of which are incorporated herein by reference.

With the widespread use of mobile terminals such as digital cameras, mobile phones and smartphones, applications and techniques for acquiring information such as the Internet, news and newspaper articles using mobile terminals and displaying them on the screen are known. Such a technique is space-saving and can easily acquire electronic information anytime and anywhere. Therefore, it improves the convenience of the user and is used for quick acquisition and utilization of information.
In the portable terminal as described above, information displayed on the screen is displayed in a horizontal or vertical screen. In addition, as a portable terminal capable of displaying information horizontally or vertically, a portable terminal capable of displaying the information by switching the screen display type such as the horizontal screen or vertical depending on the display state such as the tilt or direction of the terminal. A terminal is known (see, for example, Patent Document 1).

JP 2007-017596 A

However, for example, when a user uses a mobile terminal that can display the information by switching between a horizontal screen and a vertical screen, the screen may be switched to an unintended screen display type for the operation of the mobile terminal by the user. When such a screen display type unintended by the user occurs, the user needs to perform a new operation on the mobile terminal in order to switch to the intended screen display type. Unintentional switching of the screen display type has been a factor that reduces user convenience.

An electronic device according to one embodiment of the present invention includes a display unit that displays a display image based on display data, an imaging unit that can image a subject on the front side of the display unit, and a characteristic portion of the subject that images the front side And a display control unit that controls a display format of the display image based on a temporal change amount of the display image.

An information system according to another aspect of the present invention is an information system including an electronic device and a server, and the electronic device includes a display unit that displays a display image based on display data, and a front side of the display unit. An imaging unit capable of imaging the subject, and a communication unit that transmits a captured image by the imaging unit to the server and receives display control information for controlling a display format of the display image from the server. A communication unit that receives the captured image from the electronic device and transmits the display control information to the electronic device, and a temporal change amount of a feature portion of a subject included in the captured image received from the electronic device. And a display control information generation unit that generates the display control information to be transmitted to the electronic device.

A server according to another aspect of the present invention communicates with an electronic device including a display unit that displays a display image based on display data, an imaging unit that can capture a subject in front of the display unit, and a communication unit. A communication unit that receives the captured image from the electronic device and transmits the display control information to the electronic device; and a temporal feature of a subject portion included in the captured image received from the electronic device. And a display control information generation unit configured to generate the display control information to be transmitted to the electronic device based on a change amount.

According to the aspect of the present invention, unintended screen display type switching does not occur, and user convenience can be improved.

It is an external appearance schematic diagram of the electronic device by the 1st Embodiment of this invention. It is an external appearance schematic diagram of the electronic device by the 1st Embodiment of this invention. It is an example of the block diagram of the electronic device in this embodiment. It is an example of the block diagram in the display control part in this embodiment. It is an example of the switching control by the switching function of the display format in this embodiment. It is a mode that the electronic device in this embodiment is using. It is explanatory drawing explaining the variation | change_quantity of the to-be-photographed object's characteristic part and the to-be-photographed object's characteristic part in this embodiment. It is an example of the flowchart which shows the flow of the process at the time of the setting in this embodiment. It is an example of the flowchart which shows the flow of a display format control process in the electronic device in this embodiment. It is an example explaining the display format control processing in the present embodiment. It is an example explaining the display format control processing in the present embodiment. It is an example explaining the display format control processing in the present embodiment. It is an example explaining the display format control processing in the present embodiment. It is a flowchart which shows the other example of the display format control process in the electronic device in this embodiment. It is an example explaining the display format control processing in the present embodiment. It is an example explaining the display format control processing in the present embodiment. It is an example explaining the display format control processing in the present embodiment. It is an example explaining the display format control processing in the present embodiment. It is an example of the block diagram in the display control part with which the electronic device by the 2nd Embodiment of this invention is provided. It is a flowchart which shows an example of the display format control process in the electronic device in this embodiment. It is a flowchart which shows the other example of the display format control process in the electronic device in this embodiment. It is an example explaining the display format control processing in the present embodiment. It is an example explaining the display format control processing in the present embodiment. It is a schematic block diagram of the information system in this embodiment. It is an example of the block diagram in the display control part with which the electronic device in this embodiment is provided, the block diagram in the display control information generation part with which a server is provided, etc. It is another examples, such as a block diagram in the display control part with which the electronic device in this embodiment is provided, and a block diagram in the display control information generation part with which a server is provided.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. 1A and 1B are schematic external views of an electronic apparatus 3 (eg, a portable terminal) according to the first embodiment of the present invention. 1A shows the front surface (front surface, front surface of the main body) of the electronic device 3, and FIG. 1B shows the rear surface (back surface, rear surface of the main body) of the electronic device 3.

As shown in FIG. 1A, the electronic device 3 includes a touch screen (display unit) 44, a cursor pad 48, a front camera (lens, imaging unit) 62, a speaker 82, a microphone 84, and a determination / home button 92 on the front surface of the housing. Have In addition, as illustrated in FIG. 1B, the electronic device 3 includes a back camera (lens, imaging unit) 64 on the rear surface of the housing. In addition, as shown in FIGS. 1A and 1B, the electronic device 3 has a power button 90 on the upper part of the casing (upper part of the main body). The electronic device 3 includes a motion sensor 50 (for example, a triaxial acceleration sensor or a triaxial angular velocity sensor), a GPS 52, an azimuth meter 54, and an elevation angle sensor 56, as indicated by a broken line.

Touch screen 44 displays a display image based on the display data. The display data is, for example, image data or text data. Display images are images (for example, images such as photographs and illustrations), text (for example, characters and symbols), and the like. In addition, the touch screen 44 displays a cursor indicating a position designated by the user. The touch screen 44 includes a touch sensor and detects a touch position by the user. The display format (screen display type) of the display image displayed on the touch screen 44 is controlled by the display control unit 100 (described later) (details are described later).

The cursor pad 48 accepts a cursor movement operation displayed on the touch screen 44. That is, the cursor displayed on the touch screen 44 moves according to the movement of the user's finger on the cursor pad 48.

The front camera 62 images a subject on the front side (front side of the casing, touch screen 44 side) according to control of a camera unit 60 (described later).
The back camera 64 images a subject on the rear side (the rear side of the housing, the opposite side or the opposite side of the touch screen 44) according to control of a camera unit 60 (described later).
The front camera 62 (or the front camera 62 and the camera unit 60) is also referred to as a first imaging unit, and the back camera 64 (or the back camera 64 and the camera unit 60) is also referred to as a second imaging unit. Further, the front camera 62 and the back camera 64 may be an infrared camera or a camera with illumination so that the subject can be imaged in an environment where the subject is imaged in a dark place.

The speaker 82 outputs sound (including voice) according to control of the audio unit 80 (described later). The microphone 84 supplies the collected sound data to the audio unit 80 (described later).
The power button 90 is a button for turning on power to a casing (a casing body, a body, main body (a body, a main body)). The decision / home button 92 is a button for instructing various decisions or displaying a home screen (initial display screen).

The motion sensor 50 includes an acceleration sensor and / or an angular velocity sensor to detect movement of the casing (eg, inclination of the casing), and detects acceleration or / and angular velocity applied to the casing. The GPS 52 measures (positions) the position of the housing. The azimuth meter 54 is, for example, a geomagnetic sensor, and detects an azimuth (that is, a direction in which the housing faces). The elevation sensor 56 detects the elevation angle of the casing (for example, the angle between the incident direction of light and the ground surface, or the viewing angle from the horizontal) in order to detect the movement of the casing.

FIG. 2 is an example of a block diagram of the electronic device 3. In addition to what is shown in FIGS. 1A and 1B, the electronic device 3 includes a processor 10, an interface unit 20, a storage unit 30, an input / output unit 40, a camera unit 60, a communication unit 70, and An audio unit 80 is provided.

The processor 10 controls the entire electronic device 3. For example, the processor 10 includes the display control unit 100, and the temporal change amount (transition amount) of the feature portion of the subject imaged on the front side (that is, the feature portion on the captured image captured by the first imaging unit). The display format of the display image is controlled on the basis of (a transition amount, a transition degree), the change rate (a change rate, a transition 、 rate), and the amount of change of the characteristic portion in one period). Details of the display control unit 100 will be described later.

The interface unit 20 supplies information (for example, control information and output information) from the processor 10 to each unit. Further, the interface unit 20 supplies information (for example, stored information and input information) from each unit to the processor 10.

The storage unit 30 stores various information. For example, the storage unit 30 stores a program processed in the processor 10 and various setting information. In addition, the storage unit 30 stores display data to be displayed on the touch screen 44.

The input / output unit 40 includes a touch screen controller 42, a cursor pad controller 46, and the like. The touch screen controller 42 controls display on the touch screen 44 based on control information output from the processor 10 (display control unit 100). The cursor pad controller 46 supplies the movement operation of the cursor received via the cursor pad 48 to the processor 10.

The camera unit 60 controls the front camera 62 and the back camera 64 to image the subject. The camera unit 60 supplies the captured image to the processor 10. Note that the camera unit 60 may image the subject in accordance with the control of the processor 10.

The communication unit 70 communicates with the outside (for example, other electronic devices, servers, storages) under the control of the processor 10. The audio unit 80 controls the speaker 82 and the microphone 84 according to the control of the processor 10.

FIG. 3 is an example of a block diagram in the display control unit 100. FIG. 4 is an example of switching control by a display format switching function. FIG. 5 shows a state where the electronic device 3 is used. FIG. 6 is an explanatory diagram for explaining the feature portion of the subject and the amount of change in the subject feature portion.

As shown in FIG. 3, the display control unit 100 includes a setting recognition unit 110, an inclination amount determination unit 120, a feature region extraction unit 130, a change amount calculation unit 140, a switching determination unit 150, a switching restriction unit (hold unit) 160, A switching processing unit 180 and a setting unit 190 are provided.

The setting unit 190 sets various information. For example, ON (valid) / OFF (invalid) of the display image display format switching function is set. The setting unit 190 also sets on / off of a hold function (hold means) that is one of the display image display format control functions. The on / off of the hold function is referred to when the display image display format switching function is on. In other words, the hold function is a subfunction of the display image display format switching function. Further, the setting content by the setting unit 190 is stored in the storage unit 30.

The setting unit 190 sets the display format switching function on / off and the hold function on / off automatically or according to a user instruction according to the environment. When setting on / off by automatic setting, the setting unit 190 turns on the hold function when, for example, an application that displays a display image in a switchable display format is executed and the power saving mode is not set. Set. Note that an application that displays a display image in a switchable display format is an application that displays a display image whose display format is switched as shown below (see FIG. 4).

Hereinafter, switching control by the display format switching function will be described. 4 (a) to 4 (d), as shown in FIG. 5 (a), the user rotates the casing, and the touch screen 44 changes from a vertically oriented state (described later) to a horizontally oriented state (described later). This is an example of display image switching control in a case where the display image is changed or when the horizontal state is changed to the vertical state.
4A and 4B show switching control for rotating the display image according to the change in the orientation of the touch screen 44. FIG. For example, FIGS. 4A and 4B show an example of switching control in which the display format of the display image is switched from a portrait image (described later) to a landscape image (described later), or from a landscape image to a portrait image. 4C and 4D show switching control for reducing the display image according to the change in the orientation of the touch screen 44, that is, the display format of the display image is from a vertically long image to a horizontally long image, or from a horizontally long image. An example of switching control that does not switch to a portrait image is shown.

(Example of FIG. 4A)
The figure on the left side of the arrow in FIG. 4A is an image (hereinafter referred to as a vertically long image) in which the top and bottom direction is the longitudinal direction on the touch screen 44 in a state where the vertical direction is the longitudinal direction (hereinafter referred to as a vertically oriented state). display image HG ₁ is represents a state that is displayed. The display image HG ₁ includes text T ₁ and a photograph P ₁ . Photos _{P 1} is a portrait.

The right side of the arrow in FIG. 4A is a display on the touch screen 44 in which the horizontal direction is the longitudinal direction (hereinafter referred to as landscape orientation), which is an image in which the horizontal direction is the longitudinal direction (hereinafter landscape orientation). The image HG ₁ ′ is displayed. The display image HG ₁ ′ includes text T ₁ ′ and a photograph P ₁ . Photos _{P 1} is a portrait.

For example, when the user rotates the casing as shown in FIG. 5A and the touch screen 44 changes from the vertical orientation to the horizontal orientation (right arrow), the display format of the display image is In accordance with the change in orientation, the display image HG ₁ that is a portrait image is switched to the display image HG ₁ ′ that is a landscape image. Conversely, when the touch screen 44 is changed vertically state from landscape state (left direction arrow), the display format of the display image, the display image HG 1 _'is oblong image, the display image HG ₁ is a vertically long image And switch.
The photo P ₁ on the display image HG ₁ ′ has the same display size and display direction as the photo P ₁ on the display image HG ₁ . _'Text T ₁ of _the' on display image HG 1 is to match the a landscape display image HG 1 _', is obtained by rearranging the text T ₁ of the on the display image HG _1.

(Example of FIG. 4B)
The left side of FIG arrow in FIG. 4 (b), the touch screen 44 of the horizontally oriented state, the display image HG ₂ is a horizontally long image, and represents a state that is displayed. Display image HG ₂ is, and a text _{T 2} and the photograph _{P 2.} Photos _{P 2} is a landscape.

The diagram on the right side of the arrow in FIG. 4B shows that the display image HG ₂ ′, which is a portrait image, is displayed on the touch screen 44 in the portrait and landscape orientation. The display image HG ₂ ′ includes text T ₂ ′ and a photograph P ₂ . Photos _{P 2} is a landscape.

For example, when the user rotates the casing as shown in FIG. 5A and the touch screen 44 changes from the landscape orientation to the portrait orientation (right arrow), the display format of the display image is that of the touch screen 44. In accordance with the change in orientation, the display image HG ₂ that is a horizontally long image is switched to the display image HG ₂ ′ that is a vertically long image. Conversely, when the touch screen 44 is changed from the vertical state to the horizontal state (left direction arrow), the display format of the display image, the display image HG 2 _'is a portrait image, the display image HG ₂ is a horizontally long image And switch.
Incidentally, photographs _{P 2} on the display image HG ₂ 'are identical in photograph _{P 2} and the display size and the display direction of the display image HG _2. _'Text T ₂ of _the' on display image HG 2, as fit in a vertically long display image HG 2 _', is obtained by rearranging the text T ₂ of the on the display image HG _2.

(Example of FIG. 4C)
The left side of FIG arrow in FIG. 4 (c), similar to the left side of FIG arrow in FIG. 4 (a), the display image HG _1, represents a state that is displayed.

The diagram on the right side of the arrow in FIG. 4C shows a state in which the display image HG ₁ ″, which is a portrait image, is displayed on the touch screen 44 in the landscape orientation. The display image HG ₁ ″ includes text T ₁ ″ and a photograph P ₁ ″. The photograph P ₁ ″ is vertically long.

For example, when the user rotates the casing as shown in FIG. 5A and the touch screen 44 changes from the vertical orientation to the horizontal orientation (right arrow), the display format of the display image is according circumferential direction of the length change of the (shortened), switched from the display image HG ₁ is a vertically long image, and to reduced portrait image at a display image HG 1 _''. Conversely, when the touch screen 44 changes from the horizontal state to the vertical state (left arrow), the display format of the display image is reduced in accordance with the change in the vertical direction of the touch screen 44 (which becomes longer). The display image HG ₁ ″ which is a vertically long image is switched to the display image HG ₁ which is a vertically long image of the original size.
Note that the photo P ₁ ″ on the display image HG ₁ ″ is obtained by reducing the photo P ₁ on the display image HG ₁ so as to match the reduced display image HG ₁ ″. The same applies to the text T ₁ ″ on the display image HG ₁ ″.

(Example of FIG. 4D)
The left side of FIG arrow in FIG. 4 (d), similarly to the left side of FIG arrow in FIG. 4 (b), the display image HG _2, represents a state that is displayed.

The diagram on the right side of the arrow in FIG. 4D shows a state in which the display image HG ₂ ″, which is a landscape image, is displayed on the touch screen 44 in the portrait orientation. The display image HG ₂ ″ includes text T ₂ ″ and a photograph P ₂ ″. The photograph P ₂ ″ is horizontally long.

For example, when the user rotates the casing as shown in FIG. 5A and the touch screen 44 changes from the landscape orientation to the portrait orientation (left arrow), the display format of the display image is that of the touch screen 44. In accordance with the change (shortening) of the length in the horizontal direction, the display image HG ₂ that is a horizontally long image is switched to the display image HG ₂ ″ that is a reduced horizontally long image. Conversely, when the touch screen 44 changes from the vertical orientation to the horizontal orientation (right arrow), the display format of the display image is reduced in accordance with the change in the horizontal length of the touch screen 44 (which becomes longer). The display image HG ₂ ″ which is a horizontally long image is switched to the display image HG ₂ which is a horizontally long image of the original size.
Note that the photograph P ₂ ″ on the display image HG ₂ ″ is obtained by reducing the photograph P ₂ on the display image HG ₂ so as to match the reduced display image HG ₂ ″. The same applies to the text T ₂ ″ on the display image HG ₂ ″.

The setting recognizing unit 110 acquires setting information stored in the storage unit 30 (display image display format switching function on / off information, hold function on / off information). For example, the setting recognition unit 110 acquires the setting information stored in the storage unit 30 in response to a request from the inclination amount determination unit 120 and outputs (responds) the setting information to the inclination amount determination unit 120.

The tilt amount determination unit 120 acquires a detection value from the motion sensor 50 (elevation angle sensor 56), and calculates the tilt amount of the housing. Note that the tilt amount determination unit 120 may calculate the tilt amount of the housing when the display format switching function is set to ON.

The inclination amount determination unit 120 that calculates the inclination amount of the housing determines whether the calculated inclination amount is equal to or larger than a predetermined amount, that is, whether the housing is inclined more than the predetermined amount. The predetermined amount is, for example, 45 degrees or 60 degrees. The predetermined amount can be, for example, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 60, 65, 70, 75, 80, 85, or 90 degrees.

The tilt amount determination unit 120 does nothing when the display format switching function is set to OFF. In other words, the tilt amount determination unit 120 does not switch the display format when the display format switching function is set to OFF, so even if the tilt amount of the housing has already been calculated. ,do nothing.

In addition, when the tilt amount determination unit 120 determines that the casing is not tilted by a predetermined amount or more, the display format switching is not performed, and thus the display format switching function is set to ON. But do nothing.

In the case where the display amount switching function is set to ON and the inclination amount determination unit 120 determines that the housing is inclined by a predetermined amount or more, and the hold function of this embodiment is set to OFF. A notification (information) indicating that the hold function is set to OFF is output to the switching determination unit 150 together with the tilt amount of the housing and the tilt direction of the housing.

In the case where the display amount switching function is set to ON and it is determined that the housing is inclined by a predetermined amount or more, the inclination amount determination unit 120 is set to ON when the hold function of the present embodiment is set to ON. , The tilt amount of the casing and the tilt direction of the casing are output to the feature region extraction unit 130.

The feature region extraction unit 130 acquires a captured image captured by the front camera 62 from the camera unit 60. For example, the feature region extraction unit 130 acquires captured images before and after the tilt of the casing from the camera unit 60 when the tilt amount of the casing and the tilt direction of the casing are acquired from the tilt amount determination unit 120. That is, the feature region extraction unit 130, when the display format switching function and the hold function are set to ON, and when it is determined that the casing is tilted by a predetermined amount or more, the captured images before and after the tilt of the casing Is acquired from the camera unit 60.

The captured images before and after the tilt are a captured image before tilting the casing (a captured image immediately before starting the tilting operation or a captured image immediately after starting the tilting operation), and a captured image after tilting the casing ( (Taken image immediately after the tilting operation is completed).

Note that, among the captured images before and after the tilt, for the captured images before the tilt of the casing, the feature region extraction unit 130 acquires the tilt amount of the casing and the tilt direction of the casing from the tilt amount determination unit 120. It is an image before (past) before the time. Therefore, in the method of requesting the camera unit 60 to capture the image when the amount of inclination of the housing is acquired from the amount of inclination determination unit 120, the feature region extraction unit 130 acquires a captured image before the inclination of the housing. Is difficult. Therefore, for example, the feature region extraction unit 130 acquires a captured image before the casing is tilted by the following method.

In the first method, when the display unit switching function and the hold function are set to ON, the camera unit 60 always captures an image and temporarily stores it for a predetermined time (predetermined time). It is a method to keep. According to the second method, the camera unit 60 captures an image from the time when the housing starts to tilt by the notification from the motion sensor 50 (elevation angle sensor 56) (when the motion of the housing is detected), and is constant. This is a method of temporarily storing time (predetermined time). In any of the first method and the second method, the feature region extraction unit 130 temporarily receives the tilt amount of the casing and the tilt direction of the casing from the tilt amount determination unit 120. The pre-tilt captured image can be acquired from the images stored in the.

When the captured image before and after the tilt of the casing (the captured image before the tilt of the casing and the captured image after the tilt of the casing) is acquired from the camera unit 60, the feature region extraction unit 130 before and after the tilt of the casing, respectively. The characteristic part of the subject is extracted from the captured image. In this case, the assumed subject is a user (self) who holds the electronic device 3 and is looking at the touch screen 44. The characteristic portion is, for example, a part of the user's face (for example, eyes, ears) or a face outline.

For example, when the captured image before tilting the housing is the captured image SG illustrated in FIG. 6A and the captured image after tilting the housing is the captured image SG ′ illustrated in FIG. As illustrated in FIG. 6B, 130 extracts straight lines L ₁ and L ₁ ′ that pass through both eyes of the user from the captured image SG and the captured image SG ′ as characteristic portions of the subject.
Further, the feature region extraction unit 130, instead of the straight lines L ₁ and L ₁ ′ passing through both eyes of the user, as shown in FIG. 6C, straight lines L _{2 and} L ₂ ′ passing through both ears of the user. May be extracted as a characteristic part of the subject.
Further, instead of the above, the feature region extraction unit 130, as shown in FIG. 6 (d), the straight line L ₃ (or straight line L ₄ ), straight line L passing through the center of the rectangle adjacent to the contour of the user's face. ₃ ′ (or straight line L ₄ ′) may be extracted as a characteristic portion of the subject.

When the feature region extraction unit 130 can extract the feature portion of the subject from the respective captured images before and after the tilt of the casing, the tilt amount of the casing acquired from the tilt amount determination unit 120 and the tilt of the casing Along with the direction, information indicating the characteristic part of the subject extracted from the respective captured images before and after the tilt is output to the change amount calculation unit 140. For example, when the feature region extraction unit 130 can extract the straight lines L ₁ and L ₁ ′ shown in FIG. 6B as the feature portions of the subject, the feature region extraction unit 130 uses the slope of the straight line L ₁ as information indicating the subject feature portion. And the slope value of the straight line L ₁ ′ are output to the variation calculation unit 140.

On the other hand, the feature region extraction unit 130 acquires from the tilt amount determination unit 120 when the feature portion of the subject cannot be extracted from at least one of the captured image before tilting the housing and the captured image after tilting the housing. A notification (information) indicating that the feature portion could not be extracted is output to the switching determination unit 150 together with the amount of inclination of the casing and the inclination direction of the casing.

As shown in FIG. 1A, from the positional relationship between the touch screen 44 and the front camera 62, when the user is holding the electronic device 3 and looking at the touch screen 44, each of the cases before and after the inclination of the housing is determined. It is highly probable that a facial part or contour of the user can be extracted from the captured image.

The change amount calculation unit 140 is a subject extracted from the amount of inclination of the housing, the direction of inclination of the housing, and each captured image (the captured image before the housing is tilted and the captured image after the housing is tilted). When the information indicating the feature portion is acquired from the feature region extraction unit 130, the temporal change amount of the feature portion is calculated.

For example, the change amount calculator 140, as information indicating a characteristic portion of the object, the slope of the value of the straight line L ₁ shown in FIG. 6 (b), and, when obtaining the value of the slope of the straight line L _{1 'is} An angle θ ₁ (see FIG. 6B) formed by the straight line L ₁ and the straight line L ₁ ′ is calculated as a temporal change amount of the characteristic portion. The change amount calculation unit 140, as information indicating a characteristic portion of the object, the slope of the value of the straight line L ₂ shown in FIG. 6 (c), and, when obtaining the value of the slope of the straight line L _{2 'is} calculates the angle theta _{2 (see} FIG. 6 (c)) as a temporal change amount of the characteristic portion, the slope of the values of the straight line L ₃ shown in FIG. 6 (d), and the value of the gradient of the straight line L _{3 '} If acquired, the angle θ ₃ (see FIG. 6D) is calculated as the temporal change amount of the feature portion. The angles θ ₁ , θ ₂ , and θ ₃ described above are also the amount of rotation of the characteristic portion of the subject before and after the inclination of the housing. Incidentally, for example, the change amount calculation unit 140, when acquiring the position of the straight line L ₁ (coordinates) in the captured image shown in FIG. 6 (b) as the information indicating the characteristic portion of the object, a temporal characteristic parts It calculates a movement amount of the position of the straight line L ₁ in the captured image as the variation amount. For example, when the change amount calculation unit 140 acquires the value of the slope of the straight line L ₁ ′ with respect to the gravitational direction of the casing detected by the motion sensor 50 as information indicating the characteristic part of the subject, the time of the characteristic part is obtained. The inclination (angle) of the straight line L ₁ ′ with respect to the direction of gravity is calculated as a general change amount.

The change amount calculation unit 140 that calculates the temporal change amount of the feature portion of the subject includes the amount of inclination of the casing acquired from the feature region extraction unit 130 and the inclination direction of the casing, and the temporal change amount of the feature portion of the subject. The change amount is output to the switching determination unit 150.

The switching determination unit 150 determines whether it is necessary to switch the display format of the display image.
When the switching determination unit 150 obtains a notification (information) from the inclination amount determination unit 120 that the amount of inclination of the housing, the inclination direction of the housing, and the hold function of the present embodiment are set to off, It is determined that the display format needs to be switched.
In other words, when it is determined that the housing is inclined by a predetermined amount or more and the hold function of the present embodiment is set to OFF, the switching determination unit 150 needs to switch the display format. Is determined.

The switching determination unit 150 also displays the display format even when the amount of tilt of the housing, the tilt direction of the housing, and a notification (information) that the feature portion could not be extracted from the feature region extraction unit 130. It is determined that switching is required.
That is, when it is determined that the casing is tilted by a predetermined amount or more and the hold function is set to ON, at least the captured image before tilting the casing and the captured image after tilting the casing are at least When the feature portion of the subject is not extracted from one side (in other words, when the temporal change amount of the feature portion of the subject cannot be calculated), the switching determination unit 150 needs to switch the display format. It is determined that

In addition, when the change determination unit 150 acquires the amount of inclination of the housing, the direction of inclination of the housing, and the temporal change amount of the feature portion of the subject from the change amount calculation unit 140, the switching determination portion 150 The temporal change amount is compared with a first threshold relating to the change amount, and it is determined whether or not it is necessary to switch the display format of the display image.

For example, the switching determination unit 150 determines that the display format needs to be switched when the temporal change amount of the feature portion of the subject is equal to or greater than a first threshold, and the temporal change amount of the feature portion of the subject. Is less than the first threshold value, it is determined that it is not necessary to switch the display format of the display image.
That is, when it is determined that the housing is tilted by a predetermined amount or more and the hold function is set to ON, the temporal change amount of the feature portion of the subject is equal to or greater than the first threshold value The switching determination unit 150 determines that the display format needs to be switched (in other words, the hold processing by the hold function is unnecessary), and the temporal change amount of the feature portion of the subject is less than the first threshold value. In some cases, the switching determination unit 150 determines that switching of the display format of the display image is unnecessary (in other words, hold processing by the hold function is necessary). In one example, the display format of the display image can be controlled based on the result of detecting the change / transition of the relative posture between the subject (such as the user's face) and the housing (main body). When the change / transition of the relative posture is relatively large, the display format is switched. When the change / transition of the relative posture is relatively small, the display format is not switched.

Note that the first threshold value to be compared with the temporal change amount of the characteristic part of the subject is that when the user tilts the casing by rotating the casing, the user moves along with the tilt of the casing. It is a value for determining whether or not its own face (for example, a straight line connecting both eyes) is intentionally inclined.
An example in which the user's face is intentionally inclined with the inclination of the housing is, for example, a case as shown in FIG. On the other hand, FIG. 5A is an example in which the user does not intentionally tilt his / her face along with the tilt of the casing when the user tilts only the casing by 90 degrees.
In the case of FIG. 5A, the amount of change in the characteristic part of the subject is 90 degrees. In the case of FIG. 5B, the amount of change in the characteristic portion of the subject is approximately 0 degrees. Although not shown, when the case is inclined 90 degrees and the user is inclined 50 degrees in the same direction as the case, the amount of change in the characteristic portion of the subject is 40 degrees.
From the above, for example, the first threshold value may be an angle of about several degrees to about 45 degrees. For example, the first threshold can be 5, 10, 15, 20, 25, 30, 35, 40, or 45 degrees.
For example, when the first threshold is set to 30 degrees, when the user tilts the casing 90 degrees clockwise and the face is tilted 60 degrees or more clockwise, the amount of change is less than the first threshold. Therefore, hold processing by the hold function works.

When it is determined that the display format needs to be switched, the switching determination unit 150 outputs the tilt amount of the housing and the tilt direction of the housing to the switching processing unit 180. On the other hand, when the switching determination unit 150 determines that switching of the display format is unnecessary, the switching determination unit 150 outputs a notification (information) to the effect that the display format is maintained to the switching restriction unit 160.

When the switching restriction unit 160 obtains a notification to maintain the display format from the switching determination unit 150, the switching limitation unit 160 controls the display format of the display image so as not to be switched according to the tilt amount of the housing and the tilt direction of the housing. To do. For example, the switching restriction unit 160 outputs nothing (or a notification (information) to the effect that the display format is maintained) to the switching processing unit 180.

When the switching processing unit 180 acquires the tilt amount of the casing and the tilt direction of the casing from the switching determination unit 150, the display processing of the display image is performed according to the tilt amount of the casing and the tilt direction of the casing. Control to switch. For example, the switching processing unit 180 determines a display format (display format after switching) based on the tilt amount of the housing and the tilt direction of the housing, and outputs the determined display format to the touch screen controller 42. . As a result, the display image displayed on the touch screen 44 is switched as shown in FIG.
On the other hand, the switching processing unit 180 does nothing when the notification (information) to maintain the display format is acquired from the switching restriction unit 160.

As described above, the display control unit 100 determines that the case is tilted by a predetermined amount or more, and if the display format switching function and the hold function are set to ON, The display format of the display image on the touch screen 44 is selectively switched in accordance with the temporal change amount of the feature portion.

Note that the switching determination unit 150 may output a notification (information) indicating that the display format is maintained to the switching processing unit 180. In other words, the switching determination unit 150 may have the function of the switching restriction unit 160 (the switching restriction unit 160 may be absorbed by the switching determination unit 150).

FIG. 7 is an example of a flowchart showing a flow of processing (setting processing) related to setting of display format switching. For example, the flowchart of FIG. 7 is an example of the flow of automatic setting processing. The flowchart in FIG. 7 starts at any time (for example, periodically) when the electronic apparatus 3 is powered on. At the time of shipment, it is assumed that the storage unit 30 is set (stored) with ON / OFF of the switching control and ON / OFF of the hold function.

In FIG. 7, the processor 10 (setting recognition unit 110) refers to the storage unit 30 and determines whether or not the switching function is on (step S10).

When the switching function is on (step S10: Yes), the processor 10 (an application execution unit (not shown)) determines whether an application that displays a display image in a switchable display format is being executed ( Step S20).

In step S20, when an application that displays a display image in a switchable display format is being executed (step S20: Yes), whether or not the processor 10 (power saving mode control unit (not shown)) is in the power saving mode is determined. Is determined (step S40).

In step S40, when not in the power saving mode (energy saving mode) (step S40: No), the processor 10 (setting recognition unit 110) refers to the storage unit 30 and determines whether or not the hold function is off (step). S50). When the hold function is off (step S50: Yes), the processor 10 (setting unit 190) sets the hold function on (step S60). On the other hand, if the hold function is not off (step S50: No), that is, if the hold function is already on, step S60 is skipped, and this flowchart ends.

In step S40, when in the power saving mode (step S40: Yes), the processor 10 (setting recognition unit 110) refers to the storage unit 30 and determines whether or not the hold function is on (step S52). . When the hold function is on (step S52: Yes), the processor 10 (setting unit 190) sets the hold function to off (step S62). On the other hand, if the hold function is not on (step S52: No), that is, if the hold function is already off, step S62 is skipped, and this flowchart ends.

In step S20, when an application for displaying a display image in a switchable display format is not being executed (step S20: No), the processor 10 (setting unit 190) sets the switching function to off (step S30), and step Proceed to S62.

In step S10, when the switching function is off (step S10: No), the processor 10 (an application execution unit (not shown)) is executing an application that displays a display image in a switchable display format. Is determined (step S22).

In step S22, when the application which displays the display image of the switchable display format is being executed (step S22: Yes), the processor 10 (setting unit 190) sets the switching function to on (step S32), Proceed to step S40.
In step S22, when an application for displaying a display image in a switchable display format is not being executed (step S22: No), this flowchart ends.
The same applies to the setting processing (setting of the switching function, setting of the hold function, setting of the follow function (described later)) in the electronic device 4 according to the second embodiment of the present invention.

FIG. 8 is an example of a flowchart showing the flow of display format control processing in the electronic device 3. The flowchart in FIG. 8 starts when the housing is tilted by a predetermined amount or more when the display format switching function is set to ON. Whether or not the casing is tilted by a predetermined amount or more is determined by the tilt amount determination unit 120 as described above.

8, the setting recognition unit 110 (or the inclination amount determination unit 120) refers to the storage unit 30 and determines whether or not the hold function is on (step S110).

If the hold function is on in step S110 (step S110: Yes), the tilt amount determination unit 120 outputs the tilt amount of the housing and the tilt direction of the housing to the feature region extraction unit 130. Next, the feature region extraction unit 130 tries to extract the feature portion of the subject from each of the captured images before and after the inclination of the housing (step S120), and determines whether the feature portion of the subject has been extracted from each of them (step S120). Step S130). Note that, for example, the feature region extraction unit 130 may selectively extract the feature portion of the subject in the captured image at every predetermined time before the casing is tilted.

In step S130, when the feature portion of the subject can be extracted from each (step S130: Yes), the feature region extraction unit 130 determines the feature portion of the subject along with the amount of inclination of the housing and the direction of inclination of the housing. Is output to the change amount calculation unit 140. Next, the change amount calculation unit 140 calculates the temporal change amount of the feature portion of the subject based on the captured image (reference image) before the casing is tilted and the captured image (transition image) after the casing is tilted. In step S140, the calculated change amount is output to the switching determination unit 150 together with the inclination amount of the housing and the inclination direction of the housing. Next, the switching determination unit 150 determines whether or not the temporal change amount of the feature portion of the subject is equal to or greater than the first threshold (step S150).

If it is determined in step S150 that the temporal change amount of the characteristic portion of the subject is less than the first threshold (step S150: No), the switching determination unit 150 does not need to switch the display format of the display image. And a notification (information) to maintain the display format is output to the switching restriction unit 160. The switching restriction unit 160 that has acquired the notification (information) that the display format is maintained controls to not switch the display format of the display image, that is, to hold the switching of the display format (step S170). For example, the switching restriction unit 160 outputs a notification (information) to maintain the display format to the switching processing unit 180. The switching processing unit 180 that has acquired the notification (information) to maintain the display format from the switching restriction unit 160 does nothing. Then, this flowchart ends. Note that the rectangle in step S170 is a broken line indicates that the switching processing unit 180 does nothing.

When the hold function is not on in step S110 (step S110: No), the tilt amount determination unit 120 notifies that the hold function is set to be off together with the tilt amount of the housing and the tilt direction of the housing. (Information) is output to the switching determination unit 150. The switch determination unit 150 that has acquired the notification (information) that the hold function is set to OFF together with the tilt amount of the casing and the tilt direction of the casing determines that the display format needs to be switched. , The tilt amount of the casing and the tilt direction of the casing are output to the switching processing unit 180. The switching processing unit 180 that acquires the tilt amount of the housing and the tilt direction of the housing controls the display image display format to be switched according to the tilt amount of the housing and the tilt direction of the housing ( Step S160). For example, the switching processing unit 180 determines a display format (display format after switching) based on the tilt amount of the housing and the tilt direction of the housing, and outputs the determined display format to the touch screen controller 42. . Then, this flowchart ends.

In step S130, when the feature portion of the subject cannot be extracted from at least one of the captured images before and after the inclination of the casing (step S130: No), the feature region extraction unit 130 determines the amount of inclination of the casing, and A notification that the feature portion could not be extracted is output to the switching determination unit 150 along with the inclination direction of the housing. The switch determination unit 150 that has acquired the notification (information) that the feature portion could not be extracted together with the tilt amount of the casing and the tilt direction of the casing determines that the display format needs to be switched, and The body tilt amount and the tilt direction of the housing are output to the switching processing unit 180. Thereafter, the same processing as in the case of step S110 (No) is performed, and this flowchart ends. In addition, the switching determination unit 150 that has acquired the notification (information) that the feature portion of the subject could not be extracted sets the hold function to OFF and switches to the power saving mode described above.

If it is determined in step S150 that the temporal change amount of the feature portion of the subject is equal to or greater than the first threshold (step S150: Yes), the switching determination unit 150 needs to switch the display format of the display image. And the inclination amount of the casing and the inclination direction of the casing are output to the switching processing unit 180. Thereafter, the same processing as in the case of step S110 (No) is performed, and this flowchart ends.

FIG. 9 to FIG. 12 are examples illustrating the display format control processing. 9 to 12, it is assumed that both the display format switching function and the hold function are set to ON.

FIG. 9A shows a state where the user U is viewing the display image HG ₃ (vertically long image or vertically oriented image) displayed on the touch screen 44 in the vertically oriented state. FIG. 9B shows a captured image SG ₃ (through image) captured by the front camera 62 in the state of FIG. 9A. FIG. 9C is a display image HG ₃ (vertically long image) displayed on the touch screen 44 in the state of FIG. 9A.

FIG. 10A shows a state in which the user U tilts only the casing by a predetermined amount or more from the state of FIG. 9A. FIG. 10B shows the captured image SG ₃ ′ (through image) in the case of FIG. FIG. 10C is a display image HG ₃ ′ (landscape image or landscape image) displayed on the touch screen 44 in the case of FIG.

That is, in the flowchart of FIG. 8, the switching determination unit 150 determines that the temporal change amount of the characteristic portion of the subject is equal to or greater than the first threshold (step S150: Yes), and the switching processing unit 180 Control is performed so as to switch the display format of the display image in accordance with the amount of inclination and the inclination direction of the housing (step S160). That is, when only the casing is tilted by a predetermined amount or more as shown in FIG. 10, the portrait image is switched to the landscape image so that the user U can easily see (or as shown in FIGS. 4C and 4D). Switch to reduced image).
When the display format switching function is set to OFF, the display format is not switched.

Fig.11 (a) represents a mode that the user U inclined itself with the housing | casing more than predetermined amount from the state of Fig.9 (a). FIG. 11B shows the captured image SG ₃ (through image) in the case of FIG. FIG. 11C is a display image HG ₃ (vertically long image) displayed on the touch screen 44 in the case of FIG. The display format of the display image HG ₃ in FIG. 11C is the same as the display format of the display image HG ₃ in FIG.

That is, in the flowchart of FIG. 8, the switching determination unit 150 determines that the temporal change amount of the feature portion of the subject is less than the first threshold (step S150: No), and the switching restriction unit 160 displays the display image. The display format is controlled not to be switched, that is, the switching is held (step S170). That is, when the user and the housing are tilted by a predetermined amount or more as shown in FIG. 11, the portrait image is not switched to the landscape image so that the user U can easily see (or shown in FIGS. 4C and 4D). So do not switch to reduced image).
When the display format switching function is set to ON and the hold function is set to OFF, the display format is switched.

FIG. 12A shows a state in which only the user U tilts himself or herself by a predetermined amount or more from the state of FIG. 9A. FIG. 12B shows the captured image SG ₃ ″ (through image) in the case of FIG. FIG. 12C is a display image HG ₃ (vertically long image) displayed on the touch screen 44 in the case of FIG. The display format of the display image HG ₃ in FIG. 12C is the same as the display format of the display image HG ₃ in FIG.

That is, since the flowchart of FIG. 7 starts when the casing is tilted by a predetermined amount or more as described above, the display format is switched when only itself is tilted by a predetermined amount or more as shown in FIG. Absent.

As described above, according to the electronic device 3, since an unintended screen format (screen display type) is not switched, user convenience can be improved. For example, when the user tilts his / her face together with the housing by a predetermined amount or more as shown in FIG. 5B, the display function switching as illustrated in FIG. 4 is not caused by the hold function. User convenience can be improved.

In the above description, the electronic device 3 has described a mode in which the display format of the display image is controlled using an image captured by the front camera 62 (first imaging unit) that captures an object on the front side of the touch screen 44. However, the electronic device 3 uses the image captured by the back camera 64 (second image capturing unit) that captures the subject behind the touch screen 44 in addition to the image captured by the front camera 62 to change the display format of the display image. You may control. That is, the display control unit 100 displays the display format of the display image based on the temporal change amount of the characteristic part of the subject imaged on the front side and the temporal change amount of the characteristic part of the subject imaged on the rear side. May be controlled.

For example, the display control unit 100 is the rate of change of the characteristic part of the subject imaged on the front side (the rate of change of the characteristic part of the subject on the image captured by the front camera 62) when the casing is tilted by a predetermined amount or more. A front side change rate is calculated by calculating a front side change rate and a rear side change rate that is a change rate of the feature portion of the subject imaged on the rear side (change rate of the feature portion of the subject on the image captured by the back camera 64). Is controlled so as to change the display format of the display image, and when the forward change rate is less than the backward change rate, the display format of the display image is not changed. .

More specifically, the display control unit 100, when the housing is tilted by a predetermined amount or more and the hold function is set to ON (the display format switching function is also set to ON), the front side change rate When the forward change rate is equal to or greater than the backward change rate, control is performed to change the display format of the display image, and the forward change rate is less than the backward change rate. Controls so as not to change the display format of the display image.
On the other hand, the display control unit 100 changes the display format of the display image when the case is tilted by a predetermined amount or more and the hold function is set to off (the display format switching function is set to on). To control.

FIG. 13 is a flowchart showing another example of display format control processing in the electronic apparatus 3. For example, FIG. 13 shows a processing flow in a mode in which the display format of the display image is controlled using the image captured by the back camera 64 in addition to the image captured by the front camera 62. Note that the flowchart of FIG. 13 starts when the casing is tilted by a predetermined amount or more when the display format switching function is set to ON, as in the flowchart of FIG. Whether or not the housing is inclined by a predetermined amount or more is determined by the inclination amount determination unit 120. The same step numbers as those in FIG. 8 are the same processes as those in FIG.

In FIG. 13, the setting recognition unit 110 (or the inclination amount determination unit 120) determines whether or not the hold function is on (step S110).

If the hold function is on in step S110 (step S110: Yes), the tilt amount determination unit 120 outputs the tilt amount of the housing and the tilt direction of the housing to the feature region extraction unit 130. Next, the feature region extraction unit 130 (images taken by the front camera 62 before the tilt of the housing, images of the housing before and after the tilt of the housing) in each of the captured images (the images taken by the front camera 62 and the images taken by the back camera 64). Attempt to extract the characteristic part of the subject from the image captured by the back camera 64 before tilting the body, the image captured by the front camera 62 after tilting the casing, and the image captured by the back camera 64 after tilting the casing (step) In step S122, it is determined whether or not the feature portion of the subject has been extracted from each captured image (step S132).

In step S132, when the feature portion of the subject can be extracted from each of the captured images before and after the tilt in both captured images (step S132: Yes), the feature region extraction unit 130 determines the amount of tilt of the casing and the casing. Information indicating the characteristic part of the subject is output to the change amount calculation unit 140 along with the body tilt direction. Next, the change amount calculation unit 140 calculates temporal change rates (front-side change rate and rear-side change rate) of the characteristic portion of the subject in both captured images (step S142), and the calculated both captured images The temporal change rate of the characteristic portion is output to the switching determination unit 150 together with the amount of inclination of the casing and the inclination direction of the casing. Next, the switching determination unit 150 determines whether the front side change rate is less than the rear side change rate (step S152).

In step S152, when it is determined that the front side change rate is less than the rear side change rate (step S152: Yes), the switching determination unit 150 determines that switching of the display format of the display image is not necessary, and the display format. Is output to the switching restriction unit 160. The switching restriction unit 160 that has acquired the notification (information) that the display format is maintained controls to not switch the display format of the display image, that is, to hold the switching of the display format (step S170). Then, this flowchart ends.

When the hold function is not turned on in step S110 (step S110: No), the tilt amount determination unit 120 notifies that the hold function is set off together with the tilt amount of the housing and the tilt direction of the housing. (Information) is output to the switching determination unit 150. Next, the switching determination unit 150 determines that the display format needs to be switched, and outputs the tilt amount of the housing and the tilt direction of the housing to the switching processing unit 180. Next, the switching processing unit 180 performs control so as to switch the display format of the display image according to the tilt amount of the housing and the tilt direction of the housing (step S160). Then, this flowchart ends.

In step S132, if the feature portion of the subject cannot be extracted from any of the captured images before and after the tilt of the casing in both captured images (step S132: No), the feature region extraction unit 130 determines the tilt amount of the casing, and A notification (information) indicating that the feature portion could not be extracted is output to the switching determination unit 150 together with the inclination direction of the housing. Next, the switching determination unit 150 determines that the display format needs to be switched, and outputs the tilt amount of the housing and the tilt direction of the housing to the switching processing unit 180. Thereafter, the same processing as in the case of step S110 (No) is performed, and this flowchart ends.

When it is determined in step S152 that the front side change rate is equal to or higher than the rear side change rate (step S152: No), the switching determination unit 150 determines that the display format of the display image needs to be switched, and the housing Are output to the switching processing unit 180. Thereafter, the same processing as in the case of step S110 (No) is performed, and this flowchart ends.

14 to 17 are an example for explaining display format control processing in a mode in which the display format of the display image is controlled using the image captured by the back camera 64 in addition to the image captured by the front camera 62. 14 to 17, it is assumed that the display format switching function is set to on and the hold function is set to on.

FIG. 14A shows a state where the user U is viewing the display image HG ₄ (vertically long image) displayed on the touch screen 44 in the portrait orientation. 14 (b) is in the state of FIG. 14 (a), the captured image being captured by the front camera 62 _{SG 4-1} (through image), and the imaging is imaged by the back camera 64 images _{SG 4- 2} (through image). FIG. 14C is a display image HG ₄ (vertically long image) displayed on the touch screen 44 in the state of FIG.

FIG. 15A shows a state in which the user U tilts only the casing by a predetermined amount or more from the state of FIG. FIG. 15B illustrates a captured image SG _4-1 ′ (through image) captured by the front camera 62 and a captured image SG ₄ captured by the back camera 64 in the case of FIG. _-2 '(through image). In FIG. 15B, it is assumed that the front side change rate and the rear side change rate are the same. FIG. 15C is a display image HG ₄ ′ (horizontal image) displayed on the touch screen 44 in the case of FIG.

That is, in the flowchart of FIG. 13, the switching determination unit 150 determines that the front side change rate is equal to or higher than the rear side change rate because the front side change rate and the rear side change rate are the same (step S152: No). The switching processing unit 180 performs control so that the display format of the display image is switched according to the tilt amount of the housing and the tilt direction of the housing (step S160). That is, when only the casing is tilted by a predetermined amount or more as shown in FIG. 15, the portrait image is switched to the landscape image so that the user U can easily see (or as shown in FIGS. 4C and 4D). Switch to reduced image).
When the display format switching function is set to OFF, the display format is not switched.

FIG. 16A shows a state in which the user U tilts himself or herself with a casing by a predetermined amount or more from the state of FIG. FIG. 16 (b), in the case of FIG. 16 (a), the captured image being captured by the front camera 62 _{SG 4-1} (through image), and the imaging is imaged by the back camera 64 images _{SG 4- 2} '(through image). In FIG. 16B, the front side change rate is less than the rear side change rate. FIG. 16C is a display image HG ₄ (vertically long image) displayed on the touch screen 44 in the case of FIG. The display format of the display image HG ₄ in FIG. 16C is the same as the display format of the display image HG ₄ in FIG.

That is, in the flowchart of FIG. 13, the switching determination unit 150 determines that the forward change rate is less than the backward change rate (step S152: Yes), and the switching restriction unit 160 does not switch the display format of the display image. That is, control is performed to hold the switching (step S170). That is, when the user and the housing are tilted by a predetermined amount or more as shown in FIG. 16, the portrait image is not switched to the landscape image so that the user U can easily see (or shown in FIGS. 4C and 4D). To switch to a reduced image).
When the display format switching function is set to ON and the hold function is set to OFF, the display format is switched.

FIG. 17A shows a state in which the user U tilts only himself or more from the state of FIG. 14A by a predetermined amount or more. FIG. 17B shows a captured image SG _4-1 ″ (through image) captured by the front camera 62 and a captured image SG captured by the back camera 64 in the case of FIG. _4-2 (through image) is displayed. In FIG. 17B, the front side change rate is larger than the rear side change rate. FIG. 17C is a display image HG ₄ (vertically long image) displayed on the touch screen 44 in the case of FIG. The display format of the display image HG ₄ in FIG. 17C is the same as the display format of the display image HG ₄ in FIG.

That is, since the flowchart of FIG. 13 starts when the casing is tilted by a predetermined amount or more as described above, the display format is switched when only itself is tilted by a predetermined amount or more as shown in FIG. Absent.

As described above, even when the display format of the display image is controlled using the captured image captured by the back camera 64 in addition to the captured image captured by the front camera 62, the screen format is not changed unintentionally. Can be improved. In addition, there is an effect that the switching hold process can be performed with higher accuracy than when only the image captured by the front camera 62 is used. Further, even when the electronic device 4 is operated with the user lying down as shown in FIG. 5, the display format control according to the present embodiment does not cause unintended screen format switching. Can be improved. Therefore, the electronic device 4 according to the present embodiment is not affected by the posture when the user views the display image, and thus the display format can be controlled so that the display image is displayed in a direction that is easy for the user to visually recognize. is there.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 18 is an example of a block diagram in the display control unit 102 included in the electronic device 4 according to the second embodiment of the present invention. The appearance of the electronic device 4 according to the second embodiment of the present invention is the same as the appearance of the electronic device 3 according to the first embodiment of the present invention. The electronic device 4 is different from the electronic device 3 in that the processor 100 includes a display control unit 102 instead of the display control unit 100.

18, the display control unit 102 includes a setting recognition unit 112, an inclination amount determination unit 122, a feature region extraction unit 130, a change amount calculation unit 140, a switching control unit 170, and a setting unit 192. Since the feature region extraction unit 130 and the change amount calculation unit 140 included in the display control unit 102 are the same as the feature region extraction unit 130 and the change amount calculation unit 140 included in the display control unit 100 of the electronic device 3, a part of the description. Or omit all.

The setting unit 192 sets various information. For example, ON / OFF of a display image display format switching function is set. The setting unit 192 sets on / off of a hold function, which is one of the display image display format control functions. In addition, the setting unit 192 sets on / off of a follow function, which is one of the display image display format control functions. The hold function on / off and the follow function on / off are referred to when the display image display format switching function is on. In other words, the hold function and the follow function are subfunctions of the display image display format switching function. Further, the setting content by the setting unit 192 is stored in the storage unit 30. The following function and the following process by the following function will be described later.

The setting unit 192 sets the display format switching function on / off, the hold function on / off, and the tracking function on / off automatically or according to the user's instruction. The automatic setting of the hold function on / off and the automatic setting of the follow function on / off by the setting unit 192 are the same as the automatic setting of the hold function on / off by the setting unit 190 of the electronic device 3. is there.

The setting recognition unit 112 stores setting information stored in the storage unit 30 (display image display format switching function on / off information, hold function on / off information, tracking function on / off information). To get. For example, the setting recognition unit 112 acquires the setting information stored in the storage unit 30 in response to a request from the inclination amount determination unit 122 and outputs (responds) the setting information to the inclination amount determination unit 122.

The inclination amount determination unit 122 acquires the detection value from the motion sensor 50 (elevation angle sensor 56), and calculates the inclination amount of the casing (a casing body, a body, main body (a body, a main body)). Note that the tilt amount determination unit 122 may calculate the tilt amount of the housing when the display format switching function is set to ON.

The tilt amount determination unit 122 that calculates the tilt amount of the housing determines whether or not the housing is tilted by a predetermined amount (for example, 45 degrees or 50 degrees).

The tilt amount determination unit 122 does nothing when the display format switching function is set to OFF. That is, when the display format switching function is set to OFF, the display format is not switched, so that the tilt amount determination unit 122 has already calculated the tilt amount of the housing. ,do nothing.

When the display format switching function is set to ON and it is determined that the casing is tilted by a predetermined amount or more and the hold function is set to OFF, the tilt amount determination unit 122 Along with the amount of inclination, the direction of inclination of the case, and the result of determination related to the amount of inclination of the case (the result of determination that the amount of inclination of the case is equal to or greater than the predetermined amount) A notification (information) is output to the switching control unit 170.

When the display format switching function is set to ON and it is determined that the casing is tilted by a predetermined amount or more and the hold function is set to ON, the tilt amount determination unit 122 A determination result related to the tilt amount, the tilt direction of the housing, and the tilt amount of the housing (determination result that the tilt amount of the housing is equal to or greater than a predetermined amount) is output to the feature region extraction unit 130.

When the display format switching function is set to ON and it is determined that the casing is not tilted by a predetermined amount or more and the follow function is set to OFF, the tilt amount determination unit 122 A notification (information) indicating that the following function is set to OFF is output to the switching control unit 170 together with a determination result related to the amount of body tilt (a result of determination that the amount of tilt of the housing is less than a predetermined amount). . Note that the tilt amount determination unit 122 does not have to do anything in this case.

When the display format switching function is set to ON and it is determined that the casing is not tilted by a predetermined amount or more and the follow function is set to ON, the tilt amount determination unit 122 sets the casing. The determination result related to the body tilt amount, the tilt direction of the housing, and the tilt amount of the housing (determination result that the tilt amount of the housing is less than the predetermined amount) is output to the feature region extraction unit 130.

The feature region extraction unit 130 acquires a captured image captured by the front camera 62 from the camera unit 60. For example, the feature region extraction unit 130 determines a determination result regarding the amount of inclination of the casing, the inclination direction of the casing, and the amount of inclination of the casing (the determination result that the amount of inclination of the casing is equal to or greater than a predetermined amount, or When the inclination amount determination unit 122 acquires a determination result that the amount of inclination of the casing is less than a predetermined amount), the captured images before and after the inclination of the casing are acquired from the camera unit 60. That is, the feature region extraction unit 130 determines that the display format switching function and the hold function are set to ON and the casing is tilted by a predetermined amount or more, or the display format switching function, and When the follow function is set to ON and it is determined that the casing is not tilted by a predetermined amount or more, captured images before and after the tilt of the casing are acquired from the camera unit 60.

When the captured image before and after the tilt of the housing is acquired from the camera unit 60, the feature area extracting unit 130 extracts the feature portion of the subject from the captured images before and after the tilt of the housing. When the feature region extraction unit 130 can extract the feature portion of the subject from the respective captured images before and after the tilt of the casing, the tilt amount of the casing acquired from the tilt amount determination unit 122, the tilt direction of the casing, And the information which shows the characteristic part of the to-be-photographed object extracted from each captured image before and behind inclination with the determination result regarding the amount of inclination of a housing | casing is output to the variation calculation part 140. FIG.

On the other hand, the feature region extraction unit 130 obtains from the tilt amount determination unit 122 when the feature portion of the subject cannot be extracted from at least one of the captured image before tilting the housing and the captured image after tilting the housing. A notification (information) indicating that the feature portion could not be extracted is output to the switching control unit 170 together with the determination result relating to the tilt amount of the casing, the tilt direction of the casing, and the tilt amount of the casing.

The change amount calculation unit 140 acquires, from the feature region extraction unit 130, the determination result related to the amount of inclination of the housing, the inclination direction of the housing, the amount of inclination of the housing, and the feature portion of the subject extracted from each captured image. In such a case, the temporal change amount of the feature portion in each captured image is calculated. The change amount calculation unit 140 that calculates the temporal change amount of the feature portion of the subject determines the amount of inclination of the casing, the inclination direction of the casing, and the amount of inclination of the casing acquired from the feature region extraction unit 130. Together with the result, the temporal change amount of the characteristic part is output to the switching control unit 170.

The switching control unit 170 controls switching of the display format of the display image.
The switching control unit 170 has a determination result related to the amount of inclination of the case, the direction of inclination of the case, the amount of inclination of the case (determination result that the amount of inclination of the case is equal to or greater than a predetermined amount), and a hold function. When a notification (information) indicating that it is set to OFF is acquired from the tilt amount determination unit 122, it is determined that the display format needs to be switched.

The switching control unit 170 determines the determination result related to the tilt amount of the casing (the determination result that the tilt amount of the casing is less than the predetermined amount) and the notification (information) that the following function is set to off. Is obtained from the tilt amount determination unit 122, it is determined that the display format switching is unnecessary.

The switching control unit 170 determines the amount of inclination of the housing, the direction of inclination of the housing, the determination result related to the amount of inclination of the housing (the determination result that the amount of inclination of the housing is equal to or greater than a predetermined amount), and the characteristic portion. When a notification (information) indicating that extraction has failed is acquired from the feature region extraction unit 130, it is determined that the display format needs to be switched.

The switching control unit 170 displays the determination result related to the amount of inclination of the housing, the direction of inclination of the housing, the amount of inclination of the housing (the determination result that the amount of inclination of the housing is less than a predetermined amount), and the characteristic portion. When a notification (information) indicating that extraction has failed is acquired from the feature region extraction unit 130, it is determined that display format switching is unnecessary.

The switching control unit 170 determines the determination result related to the amount of inclination of the case, the inclination direction of the case, and the amount of inclination of the case (the determination result that the amount of inclination of the case is equal to or greater than a predetermined amount, or the case The determination result that the amount is less than the predetermined amount) and the temporal change amount of the feature portion from the change amount calculation unit 140, the temporal change amount of the feature portion and the first threshold value or The second threshold value is compared, and whether or not it is necessary to switch the display format of the display image is determined based on the comparison result relating to the change amount of the characteristic portion and the determination result relating to the inclination amount of the housing.

(When the determination result that the amount of inclination of the case is equal to or greater than the predetermined amount has been acquired)
When the temporal change amount of the feature portion of the subject is equal to or greater than the first threshold, the switching control unit 170 determines that the display format needs to be switched, and the temporal change amount of the feature portion of the subject is the first. When it is less than the threshold value of 1, it is determined that it is not necessary to switch the display format of the display image.
That is, when it is determined that the housing is tilted by a predetermined amount or more and the hold function is set to ON, the temporal change amount of the feature portion of the subject is equal to or greater than the first threshold value The switching control unit 170 determines that the display format needs to be switched (in other words, the hold process is unnecessary), and the temporal change amount of the feature portion of the subject is less than the first threshold value. The switching control unit 170 determines that switching of the display format of the display image is unnecessary (in other words, hold processing is necessary).

The first threshold value to be compared with the temporal change amount of the characteristic portion of the subject is the same as that in the first embodiment when the user tilts the housing by rotating the housing. Is a value for determining whether or not the user's own face (for example, a straight line connecting both eyes) is intentionally tilted with (in conjunction with) the tilt of the housing.

(When the judgment result that the amount of inclination of the case is less than the predetermined amount is acquired)
When the temporal change amount of the feature portion is equal to or greater than the second threshold value, the switching control unit 170 determines that the display format needs to be switched, and the temporal change amount of the feature portion is less than the second threshold value. If it is, it is determined that it is not necessary to switch the display format of the display image.
That is, when it is determined that the casing is not inclined more than the predetermined amount and the follow function is set to ON, the temporal change amount of the feature portion of the subject is equal to or larger than the second threshold value. In some cases, the switching control unit 170 determines that the display format needs to be switched (in other words, the tracking process is necessary), and the temporal change amount of the feature portion of the subject is less than the second threshold value. In this case, the switching control unit 170 determines that switching of the display format of the display image is unnecessary (in other words, no follow-up process is required).

Note that the second threshold value to be compared with the temporal change amount of the characteristic part of the subject is that when the user does not tilt the case by rotating the case, the user has his face (for example, This is a value for determining whether or not the straight line connecting both eyes is intentionally inclined.
An example in which the user's face is intentionally inclined when the casing is not inclined is, for example, a case as shown in FIG. In the case of FIG. 5C, the amount of change in the characteristic portion of the subject is 60 degrees. From the above, for example, the second threshold value may be an angle of around 45 degrees.
For example, when the second threshold is set to 50 degrees, when the user tilts only the face by 50 degrees or more without tilting the housing, the amount of change is equal to or greater than the second threshold, and the follow-up process works.

When the switching control unit 170 determines that switching of the display format is not necessary, the switching control unit 170 outputs a notification (information) to the effect that the display format is maintained to the switching processing unit 180. If the switching control unit 170 determines that the display format needs to be switched, the switching control unit 170 outputs the amount of inclination of the housing and the direction of inclination of the housing to the switching processing unit 180.

When the notification (information) indicating that the display format is maintained is acquired from the switching control unit 170, the switching processing unit 180 performs control so that the display format of the display image is not switched (for example, nothing is performed). In addition, when the switching processing unit 180 acquires the tilt amount of the casing and the tilt direction of the casing from the switching control unit 170, the switching processing unit 180 displays the display image according to the tilt amount of the casing and the tilt direction of the casing. Control to switch the display format.

As described above, the display control unit 102 determines that the case is tilted by a predetermined amount or more, and if the display format switching function and the hold function are set to ON, the subject The display format of the display image on the touch screen 44 is selectively switched in accordance with the temporal change amount of the feature portion.
In addition, the display control unit 102 determines that the case is not tilted by a predetermined amount or more, and if the display format switching function and the follow function are set to ON, the display control unit 102 The display format of the display image on the touch screen 44 is selectively switched according to the temporal change amount of the characteristic portion.

FIG. 19 is an example of a flowchart showing the flow of display format control processing in the electronic device 4. The flowchart in FIG. 19 starts when the housing is tilted (it may not be a predetermined amount or more) when the display format switching function is set to ON.

In FIG. 19, the tilt amount determination unit 122 determines whether or not the housing is tilted by a predetermined amount or more (step S200). If it is determined in step S200 that the casing is tilted by a predetermined amount or more, the following steps S210 to S270 are executed. If it is determined in step S200 that the housing is not inclined more than the predetermined amount, the following steps S310 to S370 are executed.

(S210 to step S270)
When it is determined in step S200 that the housing is tilted by a predetermined amount or more (step S200: Yes), the setting recognition unit 112 (or the tilt amount determination unit 122) refers to the storage unit 30 and the hold function is on. It is determined whether or not (step S210).

When the hold function is turned on in step S210 (step S210: Yes), the tilt amount determination unit 122 determines the determination result (the case of the tilt amount of the casing, the tilt direction of the casing, and the tilt amount of the casing). Is output to the feature area extraction unit 130. Next, the feature region extraction unit 130 tries to extract the feature portion of the subject from each of the captured images before and after the inclination of the casing (step S220), and determines whether the feature portion of the subject has been extracted from each of them (step S220) Step S230).

In step S230, when the feature portion of the subject can be extracted from each of them (step S230: Yes), the feature region extraction unit 130 determines the amount of inclination of the housing, the direction of inclination of the housing, and the amount of inclination of the housing. Together with the determination result (determination result indicating that the amount of inclination of the housing is equal to or greater than a predetermined amount), information indicating the characteristic part of the subject is output to the change amount calculation unit 140. Next, the change amount calculation unit 140 calculates a temporal change amount of the characteristic portion of the subject (step S240), and the calculated change amount is calculated based on the inclination amount of the casing, the inclination direction of the casing, and the It outputs to the switching control part 170 with the determination result which concerns on the amount of inclinations. Next, the switching control unit 170 determines whether or not the temporal change amount of the feature portion of the subject is greater than or equal to the first threshold (step S250).

If it is determined in step S250 that the temporal change amount of the characteristic portion of the subject is less than the first threshold (step S250: No), the switching control unit 170 does not need to switch the display format of the display image. And a notification (information) to maintain the display format is output to the switching processing unit 180. The switching processing unit 180 that has acquired the notification (information) to maintain the display format holds the current display format (does nothing) (step S270). Then, this flowchart ends. Note that the rectangle in step S270 is a broken line indicates that the switching processing unit 180 does not have to do anything.

When the hold function is not turned on in step S210 (step S210: No), the tilt amount determination unit 122 determines the determination result (the casing's tilt amount, the tilt direction of the casing, and the tilt amount of the casing). A determination result indicating that the tilt amount is equal to or greater than a predetermined amount) and a notification (information) indicating that the hold function is set to OFF are output to the switching control unit 170. The switching control unit 170 that has acquired the notification (information) that the hold function is set to OFF together with the determination result related to the tilt amount of the housing, the tilt direction of the housing, and the tilt amount of the housing It is determined that it is necessary to switch the format, and the amount of inclination of the casing and the inclination direction of the casing are output to the switching processing unit 180. The switching processing unit 180 that acquires the tilt amount of the housing and the tilt direction of the housing controls the display image display format to be switched according to the tilt amount of the housing and the tilt direction of the housing ( Step S260). Then, this flowchart ends.

In step S230, when the feature portion of the subject cannot be extracted from at least one of the captured images before and after the tilt of the casing (step S230: No), the feature region extraction unit 130 determines the tilt amount of the casing, the casing Switch the notification (information) that the feature portion could not be extracted, together with the determination result related to the inclination direction of the case and the determination result related to the amount of inclination of the case (determination result that the amount of inclination of the case is equal to or greater than a predetermined amount) Output to the unit 170. The switching control unit 170 that has acquired the notification (information) that the feature portion could not be extracted together with the determination result related to the tilt amount of the housing, the tilt direction of the housing, and the tilt amount of the housing, It is determined that switching is necessary, and the amount of inclination of the casing and the direction of inclination of the casing are output to the switching processing unit 180. Thereafter, the same processing as that in the case of step S210 (No) is performed, and this flowchart ends.

If it is determined in step S250 that the temporal change amount of the characteristic portion of the subject is equal to or greater than the first threshold (step S250: Yes), the switching control unit 170 needs to switch the display format of the display image. And the inclination amount of the casing and the inclination direction of the casing are output to the switching processing unit 180. Thereafter, the same processing as that in the case of step S210 (No) is performed, and this flowchart ends.

(S310 to step S370)
When it is determined in step S200 that the casing is not inclined more than a predetermined amount (step S200: No), the setting recognition unit 112 (or the inclination amount determination unit 122) refers to the storage unit 30 and the follow function is turned on. Is determined (step S310).

When the follow function is on in step S310 (step S310: Yes), the tilt amount determination unit 122 determines the determination result regarding the tilt amount of the casing, the tilt direction of the casing, and the tilt amount of the casing (the casing). Is output to the feature region extraction unit 130 as a result of determination that the inclination amount is less than a predetermined amount. Next, the feature region extraction unit 130 tries to extract the feature portion of the subject from each of the captured images before and after the inclination of the casing (step S320), and determines whether the feature portion of the subject has been extracted from each of them (step S320). Step S330).

In step S330, when the feature portion of the subject can be extracted from each of them (step S330: Yes), the feature region extraction unit 130 determines the tilt amount of the casing, the tilt direction of the casing, and the tilt amount of the casing. Together with the determination result (determination result that the amount of inclination of the housing is less than the predetermined amount) and information indicating the characteristic part of the subject are output to the change amount calculation unit 140. Next, the change amount calculation unit 140 calculates a temporal change amount of the characteristic portion of the subject (step S340), and uses the calculated change amount as a tilt amount of the casing, a tilt direction of the casing, It outputs to the switching control part 170 with the determination result which concerns on the amount of inclinations. Next, the switching control unit 170 determines whether or not the temporal change amount of the feature portion of the subject is equal to or greater than the second threshold (step S350).

If it is determined in step S350 that the temporal change amount of the feature portion of the subject is equal to or greater than the second threshold (step S350: Yes), the switching control unit 170 needs to switch the display format of the display image. And the tilt amount of the casing and the tilt direction of the casing are output to the switching processing unit 180. The switching processing unit 180 that acquires the tilt amount of the housing and the tilt direction of the housing controls the display image display format to be switched according to the tilt amount of the housing and the tilt direction of the housing ( Step S360). For example, the switching processing unit 180 determines a display format (display format after switching) based on the tilt amount of the housing and the tilt direction of the housing, and outputs the determined display format to the touch screen controller 42. . Then, this flowchart ends.

When the follow function is not turned on in step S310 (step S310: No), the tilt amount determination unit 122 determines the determination result regarding the tilt amount of the housing, the tilt direction of the housing, and the tilt amount of the housing (for the housing). A determination result indicating that the inclination amount is less than the predetermined amount) and a notification (information) indicating that the following function is set to OFF are output to the switching control unit 170. The switching control unit 170 that has acquired the notification (information) that the follow function is set to OFF together with the determination result related to the tilt amount of the casing, the tilt direction of the casing, and the tilt amount of the casing, It is determined that switching of the format is unnecessary, and a notification that the display format is maintained is output to the switching processing unit 180. The switching processing unit 180 that has acquired the notification (information) to maintain the display format does nothing (step S370). Then, this flowchart ends. Note that the rectangle in step S370 is a broken line, indicating that the switching processing unit 180 does not have to do anything.

In step S330, when the feature portion of the subject cannot be extracted from at least one of the captured images before and after the tilt of the casing (step S330: No), the feature region extraction unit 130 determines the tilt amount of the casing, the casing Switch the notification (information) that the feature portion could not be extracted, together with the determination result regarding the inclination direction of the case and the determination result related to the amount of inclination of the case (determination result that the amount of inclination of the case is less than the predetermined amount) Output to the unit 170. The switching control unit 170 that has acquired the notification (information) that the feature portion could not be extracted together with the determination result related to the tilt amount of the housing, the tilt direction of the housing, and the tilt amount of the housing, It is determined that switching is not necessary, and a notification (information) to maintain the display format is output to the switching processing unit 180. Thereafter, the same processing as in the case of step S310 (No) is performed, and this flowchart is ended.

If it is determined in step S350 that the temporal change amount of the characteristic portion of the subject is less than the second threshold (step S350: No), the switching control unit 170 does not need to switch the display format of the display image. And a notification (information) to maintain the display format is output to the switching processing unit 180. Thereafter, the same processing as in the case of step S310 (No) is performed, and this flowchart is ended. In step S270 in FIG. 19, processing in steps S222 to S270 in FIG. 20 described later may be included.

The electronic device 4 uses the image captured by the back camera 64 (second image capturing unit) that captures the subject on the rear side of the touch screen 44, in addition to the image captured by the front camera 62, similarly to the electronic device 3. The display format of the display image may be controlled.

Hereinafter, since the hold function is the same as that of the electronic apparatus 3 according to the first embodiment of the present invention, the following function will be described. The display control unit 102 calculates the front side change rate and the rear side change rate when it is not inclined by a predetermined amount or more, and when the front side change rate is equal to or higher than the rear side change rate, the display format of the display image If the front side change rate is less than the rear side change rate, control is performed to change the display format of the display image.

For example, when the case is not tilted by a predetermined amount or more and the follow function is set to ON (the display format switching function is also set to ON), the display control unit 102 When the rear side change rate is calculated and the front side change rate is equal to or greater than the rear side change rate, control is performed so as not to change the display format of the display image, and the front side change rate is less than the rear side change rate. Controls to change the display format of the display image.
On the other hand, the display control unit 102 displays the display format of the display image when the follower function is set to off (the display format switching function is set to on) when the casing is not inclined more than a predetermined amount. Control so as not to change

FIG. 20 is a flowchart showing another example of display format control processing in the electronic apparatus 4. For example, FIG. 20 illustrates a processing flow in a mode in which the display format of the display image is controlled using the image captured by the back camera 64 in addition to the image captured by the front camera 62. Note that the flowchart in FIG. 20 starts when the casing is tilted (it may not be a predetermined amount or more) when the display format switching function is set to ON, as in the flowchart in FIG. Note that the same step numbers as in FIG. 19 are the same processes as in FIG.

In FIG. 20, the tilt amount determination unit 122 determines whether or not the casing is tilted by a predetermined amount or more (step S200). If it is determined in step S200 that the casing is tilted by a predetermined amount or more, the following steps S210 to S270 are executed. If it is determined in step S200 that the housing is not inclined more than the predetermined amount, the following steps S310 to S370 are executed.

(S210 to step S270)
When it is determined in step S200 that the housing is tilted by a predetermined amount or more (step S200: Yes), the setting recognition unit 112 (or the tilt amount determination unit 122) determines whether or not the hold function is on (step). S210).

When the hold function is turned on in step S210 (step S210: Yes), the tilt amount determination unit 122 determines the determination result (the case of the tilt amount of the casing, the tilt direction of the casing, and the tilt amount of the casing). Is output to the feature area extraction unit 130. Next, the feature region extraction unit 130 (images taken by the front camera 62 before the tilt of the housing, images of the housing before and after the tilt of the housing) in each of the captured images (the images taken by the front camera 62 and the images taken by the back camera 64). Attempt to extract the characteristic part of the subject from the image captured by the back camera 64 before tilting the body, the image captured by the front camera 62 after tilting the casing, and the image captured by the back camera 64 after tilting the casing (step) In step S222, it is determined whether or not the feature portion of the subject has been extracted from each captured image (step S232).

In step S232, when the feature portion of the subject can be extracted from each of the captured images before and after the inclination in both captured images (step S232: Yes), the feature region extraction unit 130 determines the amount of inclination of the casing, Information indicating the characteristic part of the subject is output to the change amount calculation unit 140 together with the determination result related to the tilt direction and the tilt amount of the casing (the determination result that the tilt amount of the casing is equal to or greater than a predetermined amount). Next, the change amount calculation unit 140 calculates a temporal change rate (a front side change rate and a rear side change rate) of the characteristic portion of the subject in both captured images (step S242). Switch control of the calculated rate of change together with a determination result related to the tilt amount of the housing, the tilt direction of the housing, and the tilt amount of the housing (a determination result that the tilt amount of the housing is equal to or greater than a predetermined amount). Output to the unit 170. Next, the switching control unit 170 determines whether or not the front side change rate is less than the rear side change rate (step S252).

In step S252, when the forward change rate is less than the backward change rate (step S152: Yes), the switching control unit 170 determines that switching of the display format of the display image is unnecessary, and maintains the display format. A notification (information) to that effect is output to the switching processing unit 180. The switching processing unit 180 that has acquired the notification (information) to maintain the display format holds the current display format (does nothing) (step S270). Then, this flowchart ends.

When the hold function is not turned on in step S210 (step S210: No), the tilt amount determination unit 122 determines the determination result (the casing's tilt amount, the tilt direction of the casing, and the tilt amount of the casing). A determination result indicating that the tilt amount is equal to or greater than a predetermined amount) and a notification (information) indicating that the hold function is set to OFF are output to the switching control unit 170. Next, the switching control unit 170 determines that the display format needs to be switched, and outputs the tilt amount of the housing and the tilt direction of the housing to the switching processing unit 180. The switching processing unit 180 that acquires the tilt amount of the housing and the tilt direction of the housing controls the display image display format to be switched according to the tilt amount of the housing and the tilt direction of the housing ( Step S260). Then, this flowchart ends.

In step S232, when the feature portion of the subject cannot be extracted from any of the captured images before and after the tilt of the casing in both captured images (step S232: No), the feature area extraction unit 130 determines the tilt amount of the casing. In addition, the control unit switches the notification that the feature portion could not be extracted together with the determination result regarding the tilt direction of the casing and the determination result regarding the tilt amount of the casing (the determination result that the tilt amount of the casing is equal to or greater than a predetermined amount). Output to the unit 170. Next, the switching control unit 170 determines that the display format needs to be switched, and outputs the tilt amount of the housing and the tilt direction of the housing to the switching processing unit 180. Thereafter, the same processing as that in the case of step S210 (No) is performed, and this flowchart ends.

In step S252, when the front side change rate is equal to or higher than the rear side change rate (step S252: No), the switching control unit 170 determines that the display format of the display image needs to be switched, and the amount of inclination of the housing And the tilt direction of the housing is output to the switching processing unit 180. Thereafter, the same processing as that in the case of step S210 (No) is performed, and this flowchart ends.

(S310 to step S370)
When it is determined in step S200 that the housing is not inclined more than the predetermined amount (step S200: No), the setting recognition unit 112 (or the inclination amount determination unit 122) determines whether or not the following function is on. (Step S310).

When the follow function is on in step S310 (step S310: Yes), the tilt amount determination unit 122 determines the determination result regarding the tilt amount of the casing, the tilt direction of the casing, and the tilt amount of the casing (the casing). Is output to the feature region extraction unit 130 as a result of determination that the inclination amount is less than a predetermined amount. Next, the feature region extraction unit 130 tries to extract the feature portion of the subject from each of the captured images before and after the inclination in both captured images (step S322), and whether or not the feature portion of the subject has been extracted from each captured image. Is determined (step S332).

In step S332, when the feature portion of the subject can be extracted from each of the captured images before and after the inclination in both captured images (step S332: Yes), the feature region extraction unit 130 determines the amount of inclination of the casing, Information indicating the characteristic part of the subject is output to the change amount calculation unit 140 together with the determination result related to the tilt direction and the tilt amount of the casing (the determination result that the tilt amount of the casing is less than the predetermined amount). Next, the change amount calculation unit 140 calculates a temporal change rate of the characteristic part of the subject in both captured images (step S342), and uses the calculated change rate as a tilt amount of the housing, a tilt direction of the housing, and And a determination result relating to the amount of inclination of the housing (a result of determination that the amount of inclination of the housing is less than a predetermined amount) is output to the switching control unit 170. Next, the switching control unit 170 determines whether or not the front side change rate is less than the rear side change rate (step S352).

In step S352, when the front side change rate is equal to or higher than the rear side change rate (step S352: No), the switching control unit 170 determines that the display format of the display image needs to be switched, and the amount of inclination of the housing And the tilt direction of the housing is output to the switching processing unit 180. The switching processing unit 180 that acquires the tilt amount of the housing and the tilt direction of the housing controls the display image display format to be switched according to the tilt amount of the housing and the tilt direction of the housing ( Step S360). Then, this flowchart ends.

When the follow function is not turned on in step S310 (step S310: No), the tilt amount determination unit 122 determines the determination result regarding the tilt amount of the housing, the tilt direction of the housing, and the tilt amount of the housing (for the housing). A determination result indicating that the inclination amount is less than the predetermined amount) and a notification (information) indicating that the following function is set to OFF are output to the switching control unit 170. Next, the switching control unit 170 determines that switching of the display format is unnecessary, and outputs a notification (information) to the effect that the display format is maintained to the switching processing unit 180. The switching processing unit 180 that has acquired the notification to maintain the display format does nothing (step S370). Then, this flowchart ends.

In step S332, when the feature portion of the subject cannot be extracted from any of the captured images before and after the tilt of the casing in both captured images (step S332: No), the feature region extraction unit 130 determines the tilt amount of the casing. , The tilt direction of the casing, and the determination result related to the tilt amount of the casing (determination result indicating that the tilt amount of the casing is equal to or greater than the predetermined amount) and notification that the feature portion could not be extracted (information) Is output to the switching control unit 170. Next, the switching control unit 170 determines that switching of the display format is unnecessary, and outputs a notification (information) to the effect that the display format is maintained to the switching processing unit 180. Thereafter, processing similar to that in the case of step S3310 (No) is performed, and this flowchart ends.

In step S352, when the forward change rate is less than the backward change rate (step S352: Yes), the switching control unit 170 determines that switching of the display format of the display image is not necessary, and maintains the display format. A notification (information) to that effect is output to the switching processing unit 180. Thereafter, the same processing as in the case of step S310 (No) is performed, and this flowchart is ended.

21 and 22 are examples illustrating the display format control processing. For example, FIG. 21 is an example of the following function in the aspect of controlling the display format of the display image using the image captured by the front camera 62. FIG. 22 is an example of a mode following function that controls the display format of the display image using the image captured by the back camera 64 in addition to the image captured by the front camera 62. 21 and 22, it is assumed that all of the display format switching function, the hold function, and the following function are set to ON.
The hold function is the same as that of the electronic device 3 according to the first embodiment.

FIG. 21A shows a state in which only the user U is inclined by a predetermined amount or more from the state of FIG. 9A, as in FIG. FIG. 21B shows a captured image SG ₃ ″ (through image) in the case of FIG. FIG. 21C is a display image HG ₃ ′ (horizontal image) displayed on the touch screen 44 in the case of FIG.

That is, in the flowchart of FIG. 19, the switching control unit 170 determines that the temporal change amount of the feature portion of the subject is equal to or greater than the second threshold when the casing is not tilted by a predetermined amount or more (step (S350: Yes), the switching processing unit 180 performs control so as to switch the display format of the display image according to the tilt amount of the housing and the tilt direction of the housing (step S360).

FIG. 22A shows a state in which only the user U tilts himself or herself by a predetermined amount or more from the state of FIG. 22B shows a captured image SG _4-1 ″ (through image) captured by the front camera 62 and a captured image SG captured by the back camera 64 in the case of FIG. 22A. _4-2 (through image) is displayed. In FIG. 22B, the front side change rate is larger than the rear side change rate. FIG. 22C is a display image HG ₄ ′ (horizontal image) displayed on the touch screen 44 in the case of FIG.

That is, in the flowchart of FIG. 20, the switching control unit 170 determines that the front side change rate is equal to or higher than the rear side change rate when the housing is not tilted by a predetermined amount or more (step S352: No), and the switching is performed. The processing unit 180 performs control so as to switch the display format of the display image in accordance with the tilt amount of the housing and the tilt direction of the housing (step S360).

As described above, according to the electronic device 4, since an unintended screen format (screen display type) is not switched, user convenience can be improved. For example, when the user tilts his / her face together with the housing by a predetermined amount or more as shown in FIG. 5B, the display function switching as illustrated in FIG. 4 is not caused by the hold function. User convenience can be improved. Further, for example, when the user's face is tilted by a predetermined amount or more without tilting the housing, the display screen display format is switched by the following function as illustrated in FIG. Can be improved.

As described above, according to the electronic device 3 according to the first embodiment of the present invention and the electronic device 4 according to the second embodiment of the present invention, an easy-to-view display screen is provided according to the usage state of the user, and the user Improved convenience.

Note that a part of the processing of the electronic device 3 may be executed by an external device (for example, a server) to control the display format of the electronic device 3. Similarly, a part of the processing of the electronic device 4 may be executed by an external device (for example, a server) to control the display format of the electronic device 4.

FIG. 23 is a schematic configuration diagram of the information system 1. 24A and 24B are examples of a block diagram in the display control unit 104 included in the electronic device 5 and a block diagram in the display control information generation unit 200 included in the server 9. The information system 1 includes an electronic device 5 and a server 9 as shown in FIG. The electronic device 5 and the server 9 are connected by a wired or wireless network.

The electronic device 5 includes a touch screen 44, an imaging unit (a first imaging unit and a second imaging unit), and a communication unit 70, like the electronic device 3 according to the first embodiment. More specifically, the configuration of the electronic device 5 is the same as the configuration of the electronic device 3 according to the first embodiment except that the display control unit 104 is provided instead of the display control unit 100.

The display control unit 104 includes a setting unit 194 (similar to the setting unit 190 of the electronic device 3 or the setting unit 192 of the electronic device 4) as illustrated in FIG. 24A.

The communication unit 70 transmits the image captured by the imaging unit and the amount of inclination of the housing to the server 9 and receives display control information for controlling the display format of the display image on the touch screen 44 from the server 9.

The server 9 includes a communication unit 270 and a display control information generation unit 200 as shown in FIG. 24A. The display control information generation unit 200 includes a setting recognition unit 110, a tilt amount determination unit 120, a feature region extraction unit 130, a change amount calculation unit 140, and a switching determination unit 150 of the display control unit 100 of the electronic device 3 according to the first embodiment. , The switching restriction unit 160 and the switching processing unit 180 (or the setting recognition unit 112, the tilt amount determination unit 122, the feature region extraction unit 130, and the change amount calculation unit 140 of the display control unit 102 of the electronic device 4 according to the second embodiment. It has the same function as the switching control unit 170 and the switching processing unit 180).

The communication unit 270 receives the captured image and the amount of inclination of the housing from the electronic device 5. In addition, the communication unit 270 transmits display control information to the electronic device 5.

The display control information generation unit 200 generates display control information to be transmitted to the electronic device 5 based on the temporal change amount of the feature portion of the subject included in the captured image received from the electronic device 5. The display control information is information indicating the display format of the display image (corresponding to information output from the switching processing unit 180 of the electronic device 3 (electronic device 4) to the touch screen controller 42).

Note that the setting information (for example, hold function on / off information) of the electronic device 5 may be stored in the server 9 or may be stored in the storage unit 30. The display control information generation unit 200 of the server 9 receives setting information from the electronic device 5 as necessary, and refers to it when generating display control information.

Note that the display control unit 104 of the electronic device 5 includes a setting recognition unit 110 (112), an inclination amount determination unit 120 (122), and a feature region extraction unit 130 in addition to the setting unit 194, as shown in FIG. 24B. May be. In the above case, the display control information generation unit 200 of the server 9 does not include the setting recognition unit 110 (112), the inclination amount determination unit 120 (122), and the feature region extraction unit 130 as illustrated in FIG. 24B. Also good.

In the case where the electronic device 5 includes the feature region extraction unit 130 and the like, the electronic device 5 has the processing result (for example, the hold function is set to off) instead of the captured image. Notification) or processing result of the feature area extraction unit 130 (information indicating the feature portion of the subject or notification that the feature portion could not be extracted) is transmitted to the server 9. Further, the display control information generation unit 200 of the server 9 generates display control information using a processing result acquired from the electronic device 5 instead of the captured image.

Note that in the electronic device 3 (the same applies to the electronic devices 4 and 5), when the display format of the display image is not controlled using the image captured by the back camera 64, the electronic device 3 does not have the back camera 64. May be.

In the electronic device 3 (same for the server 9), the hold process may be performed only on the face stored in advance. That is, the feature of the subject extracted by the feature region extraction unit 130 from the captured image captured by the front camera 62 when the display format switching function and the hold function are set to ON and the casing is tilted by a predetermined amount or more. When it is determined that the portion matches the pre-stored facial feature portion, the hold process is executed (that is, the display format is not switched), and the feature portion is stored in the pre-stored facial feature portion. When it is determined that the feature portion does not match, the hold process may not be executed (that is, the display format may be switched),
Similarly, in the electronic device 4 (same for the server 9), the following process may be performed only on the face stored in advance.

Note that a program for executing processing of the electronic device 3 and the electronic device 4 according to the embodiment of the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system, By executing, the above-described various processes related to the electronic device 3 and the electronic device 4 according to the embodiment of the present invention may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” refers to a floppy (registered trademark) disk, a magneto-optical disk, an SD card, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, and a computer system. A built-in storage device such as a hard disk.

Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes a design and the like within the scope not departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 ...

Information system

3, 4, 5 ... Electronic device 9 ... Server 10 ... Processor 20 ... Interface part 30 ... Memory | storage part 40 ... Input / output part 42 ... Touch screen controller 44 ... Touch screen 46 ... Cursor pad controller 48 ... Cursor pad 50 ... motion sensor 52 ... GPS 54 ... azimuth meter 56 ... elevation sensor 60 ... camera part 62 ... front camera 64 ... back camera 70 ... communication part 80 ... audio part 82 ... speaker 84 ... microphone 90 ... power button 92 ... decision / home button DESCRIPTION OF

SYMBOLS

100, 102, 104 ...

Display control part

110, 112 ... Setting

recognition part

120, 122 ... Inclination amount determination part 130 ... Feature area extraction part 140 ... Change amount calculation part 150 ... Switching determination part 160 ... Switching restriction part ( Rudo portion) 170 ... the switching control unit 180 ... switching

unit

190, 192 ... setting unit 200 ... display control information generating section 270 ... communication unit

Claims

A display unit for displaying a display image based on the display data;
An imaging unit capable of imaging a subject on the front side of the display unit;
An electronic apparatus comprising: a display control unit that controls a display format of the display image based on a temporal change amount of a characteristic portion of a subject that images the front side.
The electronic device according to claim 1,
The display control unit
An electronic apparatus, wherein the display format of the display image on the display unit is selectively switched according to the amount of change.
The electronic device according to claim 1 or 2,
The electronic apparatus, wherein the imaging unit starts imaging the subject when detecting a movement of a housing.
The electronic device according to any one of claims 1 to 3,
The electronic device according to claim 1, wherein the display control unit controls the display format based on a temporal change amount of the characteristic portion from a time when the movement of the housing is detected to a predetermined time.
The electronic device according to any one of claims 1 to 4,
A sensor for detecting the inclination of the housing;
The display control unit
When the case is inclined more than a predetermined amount,
When the amount of change is equal to or greater than a first threshold, control to change the display format of the display image,
When the amount of change is less than a first threshold, control is performed so as not to change the display format of the display image.
The electronic device according to any one of claims 1 to 4,
A sensor for detecting the inclination of the housing;
A setting unit for setting on / off of a hold function which is a control function of the display format,
The display control unit
When the case is tilted more than a predetermined amount and the hold function is set to ON,
When the amount of change is equal to or greater than a first threshold, control to change the display format of the display image,
When the amount of change is less than the first threshold, control so as not to change the display format of the display image,
When the case is tilted more than a predetermined amount and the hold function is set to off,
An electronic apparatus characterized by controlling to change the display format of the display image.
The electronic device according to any one of claims 1 to 4,
A sensor for detecting the inclination of the housing;
The display control unit
When the casing is not inclined more than a predetermined amount,
When the amount of change is less than the second threshold, control to not change the display format of the display image,
When the change amount is equal to or larger than a second threshold value, the electronic apparatus is controlled to change a display format of the display image.
The electronic device according to any one of claims 1 to 4,
A sensor for detecting the inclination of the housing;
A setting unit for setting on / off of a follow function that is a control function of the display format;
The display control unit
When the case is not inclined more than a predetermined amount and the follow function is set to ON,
When the change amount is less than the second threshold value, control is performed so as not to change the display format of the display image, and when the change amount is greater than or equal to the second threshold value, the display format of the display image is changed. Control and
When the case is not inclined more than a predetermined amount and the follow function is set to off,
An electronic apparatus, wherein control is performed so as not to change a display format of the display image.
The electronic device according to any one of claims 1 to 8,
A second imaging unit that images a subject on the rear side of the display unit;
The display control unit
Controlling the display format of the display image based on the temporal change amount of the feature portion of the subject imaged on the front side and the temporal change amount of the feature portion of the subject imaged on the rear side. Features electronic equipment.
The electronic device according to claim 9,
A sensor for detecting the inclination of the housing;
The display control unit
When the case is inclined more than a predetermined amount,
Calculating a front side change rate that is a change rate of a feature portion of a subject that images the front side and a rear side change rate that is a change rate of a feature portion of a subject that images the back side;
When the front side change rate is equal to or higher than the rear side change rate, control to change the display format of the display image,
When the front side change rate is less than the rear side change rate, control is performed so as not to change the display format of the display image.
The electronic device according to claim 9,
A sensor for detecting the inclination of the housing;
A setting unit for setting on / off of a hold function which is a control function of the display format,
The display control unit
When the case is tilted more than a predetermined amount and the hold function is set to ON,
Calculating a front side change rate that is a change rate of a feature portion of a subject that images the front side and a rear side change rate that is a change rate of a feature portion of a subject that images the back side;
When the front side change rate is equal to or higher than the rear side change rate, control to change the display format of the display image,
When the front side change rate is less than the rear side change rate, control so as not to change the display format of the display image,
When the case is tilted more than a predetermined amount and the hold function is set to off,
An electronic apparatus characterized by controlling to change the display format of the display image.
The electronic device according to claim 9,
A sensor for detecting the inclination of the housing;
The display control unit
When the casing is not inclined more than a predetermined amount,
Calculating a front side change rate that is a change rate of a feature portion of a subject that images the front side and a rear side change rate that is a change rate of a feature portion of a subject that images the back side;
When the front side change rate is equal to or higher than the rear side change rate, control to change the display format of the display image,
When the front side change rate is less than the rear side change rate, control is performed so as not to change the display format of the display image.
The electronic device according to claim 9,
A sensor for detecting the inclination of the housing;
A setting unit for setting on / off of a follow function that is a control function of the display format;
The display control unit
When the case is not inclined more than a predetermined amount and the follow function is set to ON,
Calculating a front side change rate that is a change rate of a feature portion of a subject that images the front side and a rear side change rate that is a change rate of a feature portion of a subject that images the back side;
When the front side change rate is equal to or higher than the rear side change rate, control is performed so as to change the display format of the display image, and when the front side change rate is less than the rear side change rate, the display image is displayed. Control the display format of
When the case is not inclined more than a predetermined amount and the follow function is set to off,
An electronic apparatus, wherein control is performed so as not to change a display format of the display image.
The electronic device according to claim 6 or 11,
The setting unit
An electronic device, wherein the hold function is set to ON when the power saving mode is not set.
The electronic device according to claim 8 or 13,
The setting unit
An electronic device, wherein the follow function is turned on when the power saving mode is not set.
An information system including an electronic device and a server,
The electronic device is
A display unit for displaying a display image based on the display data;
An imaging unit capable of imaging a subject on the front side of the display unit;
A communication unit that transmits a captured image by the imaging unit to the server and receives display control information for controlling a display format of the display image from the server;
The server
A communication unit that receives the captured image from the electronic device and transmits the display control information to the electronic device;
A display control information generation unit configured to generate the display control information to be transmitted to the electronic device based on a temporal change amount of a characteristic portion of a subject included in a captured image received from the electronic device. Information system.
A server that communicates with an electronic device that includes a display unit that displays a display image based on display data, an imaging unit that can image a subject on the front side of the display unit, and a communication unit,
A communication unit that receives the captured image from the electronic device and transmits the display control information to the electronic device;
A display control information generation unit configured to generate the display control information to be transmitted to the electronic device based on a temporal change amount of a characteristic portion of a subject included in a captured image received from the electronic device. Server.