KR20130006154A - Method for screen orientation determination and mobile terminal using this method - Google Patents

Abstract

PURPOSE: Screen orientation determination method and mobile terminal using this are provided to recognize face orientation based on image recognition, decide screen orientation to match with face orientation, and change screen orientation automatically, thereby decide the screen orientation by minimizing user inconvenience. The screen direction is decided while the inconvenience of the user is minimized. CONSTITUTION: Sensor(140) senses inclination of body. Control unit(180) recognizes face orientation through camera(121), and decides screen orientation according to inclination of the body and orientation of the face. Output unit(150) indicates screen by the determined orientation. The sensor senses the orientation using gravity sensor, gyro sensor or any combinations. The control unit recognizes face or position of specific part of the face. [Reference numerals] (110) Wireless communication unit; (111) Broadcasting reception module; (112) Mobile communication module; (113) Wireless internet module; (114) Local area communication module; (115) Location information module; (120) A/V input unit; (121) Camera; (122) Microphone; (130) User input unit; (140) Sensor; (141) Proximity sensor; (150) Output unit; (151) Display unit; (152) Sound output module; (153) Alarm unit; (154) Haptic module; (160) Memory; (170) Interface module; (180) Control unit; (181) Multimedia module; (190) Power supply unit

Description

Method for determining screen orientation and mobile terminal using same {METHOD FOR SCREEN ORIENTATION DETERMINATION AND MOBILE TERMINAL USING THIS METHOD}

The present disclosure relates to a mobile terminal, and more particularly, to a screen direction determination method and a mobile terminal using the same.

Terminals may be divided into mobile / portable terminals and stationary terminals according to their mobility. The mobile terminal may be further classified into a handheld terminal and a vehicle mount terminal according to whether a user can directly carry it.

As the functions of such terminals are diversified, they are implemented in the form of a multimedia player having complex functions such as photographing and photographing of movies, reproduction of music and video files, games, and broadcasting reception . In order to support and enhance the functionality of such terminals, it may be considered to improve the structural and / or software parts of the terminal.

Recently, with the development of mobile terminal technology, a mobile terminal (or smart phone) employing a user interface (UI) in consideration of user convenience has been released. In particular, a technology for determining a screen of a mobile terminal has been developed and utilized to easily check a long message by determining a display screen display direction from a portrait direction to a landscape direction as needed.

However, in the case of an existing mobile terminal, when the screen orientation of the mobile terminal is changed during use (for example, when watching a movie while lying down), the screen does not match the intention of the user. There has been an inconvenience of having to turn the screen a few times again. Therefore, there is a need for an effective method that can conveniently provide the user with the screen orientation of the mobile terminal.

Accordingly, an object of the present specification is to provide an apparatus and method for determining a screen orientation while minimizing user inconvenience.

According to an embodiment of the present disclosure, a method of changing a screen orientation of a mobile terminal may include: detecting, by a terminal, an inclination of a body, recognizing a direction of a face through a camera, according to the detected inclination of the body and a direction of a face; The method may include determining a screen orientation of the screen and displaying the screen in the determined direction.

       The detecting of the inclination of the terminal body may include detecting an inclination degree based on a body that is in a horizontal direction with the ground using an acceleration sensor, a gyro sensor, a geomagnetic sensor, or any combination thereof.

Recognizing the direction of the face may be characterized by recognizing the direction of the face or a specific portion of the face. Recognizing the direction of the face may be characterized in that the direction of the face is identified in the image of the face and the direction is parallel to the specific position of the short direction of the face. Recognizing the direction of the specific part of the face may be characterized by using the direction of the two eyes in the image of the face, the direction and the horizontal direction of the identified two eyes.

Recognizing the direction of the face may be characterized in that the camera recognizes the image of the face at a predetermined time interval. Recognizing the direction of the face may be characterized in that selectively performed by the mode.

The determining of the direction of the screen may include comparing the screen direction generated in the horizontal or vertical view state of the set screen with the direction of the face recognized from the image of the user face or the direction of a specific part of the face. Can be.

The determining of the direction of the screen may include changing the set horizontal view state to the vertical view state if the inclination between the screen direction and the direction of the face or a specific part of the face does not fall within a predetermined range according to the comparison result. It may be characterized by changing the view state to the landscape view state.

The determining of the direction of the screen may include maintaining the set horizontal or vertical view state when the inclination with respect to each other between the screen direction and the direction of the face or a specific part of the face falls within a predetermined range according to the comparison result. It can be characterized.

As described above, according to the screen orientation determining method and the mobile terminal using the same according to an embodiment disclosed herein, the mobile terminal recognizes the direction of the user's face using image recognition, and adjusts the screen orientation to match the recognized face direction. By determining whether to make a decision and automatically changing the screen orientation if necessary, it is possible to perform screen orientation determination while minimizing user inconvenience.

1 is a block diagram illustrating a configuration of a method for determining a screen direction using image recognition of a mobile terminal and a mobile terminal according to an exemplary embodiment disclosed herein.
2 is a flowchart illustrating a method of determining a screen direction according to a direction of a face using image recognition of a mobile terminal according to one embodiment disclosed herein.
3 is an exemplary diagram illustrating a method of determining a screen direction according to a direction of a face through image recognition according to an exemplary embodiment disclosed herein.
4 is an exemplary diagram illustrating a method of determining a screen direction according to a direction of a specific part of a face through image recognition according to an exemplary embodiment disclosed herein.
FIG. 5 is a flowchart illustrating a method of determining a screen orientation using image recognition through mode selection by a mobile terminal according to an exemplary embodiment disclosed herein.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments disclosed herein will be described in detail with reference to the accompanying drawings, and the same or similar components will be given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. In addition, it should be noted that the attached drawings are only for easy understanding of the embodiments disclosed in the present specification, and should not be construed as limiting the technical idea disclosed in the present specification by the attached drawings.

Overall configuration of the mobile terminal

The mobile terminal described herein may include a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant, a portable multimedia player (PMP), navigation, and the like. However, it will be readily apparent to those skilled in the art that the configuration according to the embodiments described herein may also be applied to fixed terminals such as digital TVs, desktop computers, etc., except when applicable only to mobile terminals.

1 is a block diagram illustrating a configuration of a method for determining a screen direction using image recognition of a mobile terminal and a mobile terminal according to an exemplary embodiment disclosed herein.

The mobile terminal 100 includes a wireless communication unit 110, an audio / video input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, A controller 170, a controller 180, a power supply 190, and the like. The components shown in FIG. 1 are not essential, and a mobile terminal having more or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more modules for enabling wireless communication between the mobile terminal 100 and the wireless communication system or between the mobile terminal 100 and the network in which the mobile terminal 100 is located. For example, the wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short range communication module 114, and a location information module 115 .

The broadcast receiving module 111 receives a broadcast signal and / or broadcast related information from an external broadcast management server through a broadcast channel. The broadcast-related information may refer to a broadcast channel, a broadcast program, or information related to a broadcast service provider. The broadcast related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 112. The broadcast signal and / or broadcast related information received through the broadcast receiving module 111 may be stored in the memory 160.

 The mobile communication module 112 transmits and receives radio signals to at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and may be built in or externally attached to the mobile terminal 100. Wireless Internet technologies may include Wireless LAN (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), and the like.

The short range communication module 114 refers to a module for short range communication. Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like can be used as a short range communication technology.

The location information module 115 is a module for receiving a location of a mobile terminal, and a representative example thereof is a GPS (Global Position System) module.

Referring to FIG. 1, an A / V (Audio / Video) input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video call mode or the photographing mode. The processed image frame can be displayed on the display unit 151. [

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. [ Two or more cameras 121 may be provided depending on the use environment.

The microphone 122 receives an external sound signal by a microphone in a call mode, a recording mode, a voice recognition mode, etc., and processes the external sound signal into electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 112 when the voice data is in the call mode, and output. Various noise reduction algorithms may be implemented in the microphone 122 to remove noise generated in receiving an external sound signal.

The user input unit 130 generates input data for a user to control the operation of the terminal. The user input unit 130 may include a key pad dome switch, a touch pad (static pressure / capacitance), a jog wheel, a jog switch, and the like.

The sensing unit 140 senses the current state of the mobile terminal 100 such as the open / close state of the mobile terminal 100, the position of the mobile terminal 100, the presence or absence of user contact, the orientation of the mobile terminal, And generates a sensing signal for controlling the operation of the mobile terminal 100. For example, when the mobile terminal 100 is in the form of a slide phone, it may sense whether the slide phone is opened or closed. In addition, whether the power supply unit 190 is supplied with power, whether the interface unit 170 is coupled to the external device may be sensed. Meanwhile, the sensing unit 140 may include a proximity sensor 141.

The output unit 150 may include a display unit 151, an audio output module 152, an alarm unit 153, and a haptic module 154, for example, for generating output related to visual, auditory, have.

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, when the mobile terminal is in the call mode, a UI (User Interface) or a GUI (Graphic User Interface) associated with a call is displayed. When the mobile terminal 100 is in the video communication mode or the photographing mode, the photographed and / or received video or UI and GUI are displayed.

The display unit 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible). display, a 3D display, or an e-ink display.

Some of these displays may be transparent or light transmissive so that they can be seen through. This can be referred to as a transparent display, and a typical example of the transparent display is TOLED (Transparent OLED) and the like. The rear structure of the display unit 151 may also be of a light transmission type. With this structure, the user can see an object located behind the terminal body through the area occupied by the display unit 151 of the terminal body.

There may be two or more display units 151 according to the embodiment of the mobile terminal 100. For example, in the mobile terminal 100, a plurality of display portions may be spaced apart from one another, or may be disposed integrally with one another, and may be disposed on different surfaces, respectively.

(Hereinafter, referred to as a 'touch screen') in which a display unit 151 and a sensor for sensing a touch operation (hereinafter, referred to as 'touch sensor') form a mutual layer structure, It can also be used as an input device. The touch sensor may have the form of, for example, a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in a pressure applied to a specific portion of the display unit 151 or a capacitance generated in a specific portion of the display unit 151 into an electrical input signal. The touch sensor can be configured to detect not only the position and area to be touched but also the pressure at the time of touch.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the controller 180. As a result, the controller 180 can know which area of the display unit 151 is touched.

Referring to FIG. 1, a proximity sensor 141 may be disposed in an inner region of the mobile terminal or in the vicinity of the touch screen, which is surrounded by the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or a nearby object without mechanical contact using the force of an electromagnetic field or infrared rays. The proximity sensor has a longer life span than the contact sensor and its utilization is also high.

Examples of the proximity sensor include a transmission photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. And to detect the proximity of the pointer by the change of the electric field along the proximity of the pointer when the touch screen is electrostatic. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

The sound output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. The sound output module 152 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, etc.) performed in the mobile terminal 100. [ The audio output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying the occurrence of an event of the mobile terminal 100. Examples of events that occur in the mobile terminal include call signal reception, message reception, key signal input, touch input, and the like. The alarm unit 153 may output a signal for notifying the occurrence of an event in a form other than the video signal or the audio signal, for example, vibration. The video signal or the audio signal may be output through the display unit 151 or the audio output module 152 so that they may be classified as a part of the alarm unit 153.

The haptic module 154 generates various tactile effects that the user can feel. Vibration is a representative example of the haptic effect generated by the haptic module 154. The intensity and pattern of vibration generated by the haptic module 154 can be controlled. For example, different vibrations may be synthesized and output or sequentially output.

In addition to vibration, the haptic module 154 may be configured to provide a pin array that vertically moves with respect to the contact skin surface, a jetting force or suction force of air through an injection or inlet port, grazing to the skin surface, contact of an electrode, electrostatic force, and the like. Various tactile effects can be generated, such as effects by the endothermic and the reproduction of a sense of cold using the elements capable of endotherm or heat generation.

The haptic module 154 can be implemented not only to transmit the tactile effect through the direct contact but also to allow the user to feel the tactile effect through the muscular sensation of the finger or arm. At least two haptic modules 154 may be provided according to the configuration of the mobile terminal 100.

 The memory 160 may store a program for the operation of the controller 180 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 160 may store data on vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 160 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic It may include a storage medium of at least one type of disk, optical disk. The mobile terminal 100 may operate in association with a web storage that performs a storage function of the memory 160 on the Internet.

The interface unit 170 serves as a path for communication with all external devices connected to the mobile terminal 100. The interface unit 170 receives data from an external device, receives power, transfers the power to each component inside the mobile terminal 100, or transmits data inside the mobile terminal 100 to an external device. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, an audio I / O port, A video input / output (I / O) port, an earphone port, and the like may be included in the interface unit 170.

The identification module is a chip for storing various information for authenticating the usage right of the mobile terminal 100 and includes a user identification module (UIM), a subscriber identity module (SIM), a general user authentication module a universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the port.

The interface unit may be a passage through which power from the cradle is supplied to the mobile terminal 100 when the mobile terminal 100 is connected to an external cradle, or various command signals input from the cradle by a user may be transferred. It may be a passage that is delivered to the terminal. The various command signals or the power source input from the cradle may be operated as a signal for recognizing that the mobile terminal is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the mobile terminal. For example, voice communication, data communication, video communication, and the like. The control unit 180 may include a multimedia module 181 for multimedia playback. The multimedia module 181 may be implemented in the control unit 180 or may be implemented separately from the control unit 180. [

The controller 180 may perform a pattern recognition process for recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components.

The various embodiments described herein may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein may be implemented as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays , Microprocessors, microprocessors, microprocessors, and other electronic units for carrying out other functions. In some cases, the embodiments described herein may be implemented by the controller 180 itself.

According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code can be implemented in a software application written in a suitable programming language. The software code is stored in the memory 160 and can be executed by the control unit 180. [

How to handle user input to a mobile terminal

The user input unit 130 is operated to receive a command for controlling the operation of the mobile terminal 100 and may include a plurality of operation units. The manipulation units may also be referred to collectively as a manipulating portion, and may be employed in any manner as long as the user is tactile in the manner of tactile feeling.

Various types of time information can be displayed on the display unit 151. [ These pieces of information can be displayed in the form of letters, numbers, symbols, graphics, or icons.

At least one of the letters, numbers, symbols, graphics, or icons may be displayed in a predetermined arrangement for inputting such information, thereby being implemented as a keypad. Such a keypad may be called a so-called " soft key ".

The display unit 151 may operate as an entire area or may be divided into a plurality of areas and operated. In the latter case, the plurality of areas can be configured to operate in association with each other.

For example, an output window and an input window may be displayed on the upper and lower portions of the display unit 151, respectively. The output window and the input window are areas allocated for outputting or inputting information, respectively. In the input window, a softkey displaying a number for inputting a telephone number or the like may be output. When the softkey is touched, a number or the like corresponding to the touched softkey is displayed in the output window. When the operation unit is operated, call connection to the telephone number displayed in the output window may be attempted or text displayed in the output window may be input to the application.

The display unit 151 or the touch pad may be configured to receive a touch input by scrolling. The user may move a cursor or a pointer located on an object displayed on the display unit 151, for example, an icon, by scrolling the display unit 151 or the touch pad. Further, when the finger is moved on the display unit 151 or the touch pad, the path along which the finger moves may be visually displayed on the display unit 151. [ This will be useful for editing the image displayed on the display unit 151. [

In response to the display unit 151 (touch screen) and the touch pad being touched together within a predetermined time range, one function of the terminal may be executed. In the case of being touched together, there may be a case where the user clamps the terminal body using the thumb and index finger. For example, the function may be activation or deactivation of the display unit 151 or the touch pad.

Hereinafter, embodiments related to a control method that can be implemented in the terminal configured as described above will be described with reference to the accompanying drawings. Embodiments described later may be used alone or in combination with each other. In addition, embodiments described below may be used in combination with the above-described user interface (UI).

Screen direction determination method and mobile terminal using same

In the case of the existing mobile terminal (or smart phone), the screen direction is screened using various sensors (accelerometer sensor, gyroscope sensor, geomagnetic sensor, global positioning system) Collect the information needed for Orientation. However, when using the conventional method, there was a problem that occurs when the user is inconsistent with the desired screen orientation.

When the mobile terminal 100 (for example, a smart phone) according to the exemplary embodiment disclosed in the present specification is to determine the screen direction, the mobile terminal 100 recognizes the direction of the user's face, that is, the direction of the face or a specific part of the face using image recognition. Also, by determining whether the screen direction is determined to match the recognized face direction and automatically changing the screen direction if necessary, the screen direction determination can be performed while minimizing user inconvenience.

For example, when the screen orientation changes, the camera recognizes the position of the user's eyes and face, and based on the information, the direction of the screen does not match the direction in which the face and eyes are located, that is, the current screen. When the direction of is perpendicular to both eyes of the user, the screen orientation of the terminal may be automatically changed (or corrected).

As shown in FIG. 1, the apparatus for determining the screen orientation using image recognition of the mobile terminal 100 may include a wireless communication unit 110, an A / V input unit 120, a user input unit 130, a sensing unit 140, and a controller ( 180 and memory 160. The A / V input unit 120 may include a camera module 121 for image recognition, and the sensing unit 140 may include an acceleration sensor, a geomagnetic sensor, and a GPS sensor, and the controller 180 may display a screen. It may include a recognition module and a screen orientation determination module.

Here, the acceleration sensor is a device for measuring the acceleration of the terminal and measures how much the speed has changed in a certain period of time. The terminal including the acceleration sensor may determine whether the speed has risen or the speed has fallen. Gyro sensors are sensors that detect acceleration. This sensor measures how much the angle moves in one second. Accelerometer sensor is basically used as a sensor for compensating the gyro. The acceleration sensor indicates the acceleration of each of the X, Y, and Z axes and the inclination in each direction, and the gyro sensor indicates how many degrees the sensor is rotated.

Geomagnetic sensor is a sensor that can measure the strength of the magnetic field in the earth. It is a sum of the output vectors between X, Y, and Z axis sensors that can measure the strength of the magnetic field. Can be determined. By setting the zero degree direction, the gyro sensor can determine the current direction by how many degrees it has rotated from that direction.

But first you need to determine where the map is at 0 degrees. In other words, to write on the map, you need to know where the north is. In order to know the direction of the map, geomagnetic sensor or GPS should be used to match the direction of North once and then the direction should be set to 0 degree. In particular, the gyro sensor loses its zero point as it continues to rotate, resulting in a lot of errors, which can be changed with a geomagnetic sensor.

The sensing unit 140 may detect an inclination degree based on a direction in which the body of the terminal is horizontal to the ground using an acceleration sensor, a gyro sensor, a geomagnetic sensor, or any combination thereof.

The screen orientation recognition module of the controller 180 reproduces the screen obtained from the wireless communication unit 110, the A / V input unit 120, the memory 160, and the like according to the inclination of the terminal detected by the sensing unit 140. It can perform the function to recognize the direction of the screen.

The screen direction determination module of the controller 180 recognizes the screen direction recognized by the screen direction recognition module of the controller 180 and the face direction of the user recognized through the camera, that is, the direction of the face or the face specific part. If the screen orientation is determined to match the recognized facial orientation, the screen orientation can be automatically changed if necessary.

2 is a flowchart illustrating a method of determining a screen direction according to a direction of a face using image recognition of a mobile terminal according to one embodiment disclosed herein.

As shown in FIG. 2, the method for determining the screen direction according to the direction of the face using the image recognition by the mobile terminal 100 includes a tilt detection step (S200), a screen direction obtaining step (S210), and a face direction obtaining step (S220). , Comparing the screen direction with the face direction (S230), determining (S240), changing the screen direction (S250), and maintaining the screen direction (260).

In the tilt detection step (S200), the sensing unit 140 uses a variety of sensors, such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor, to determine the inclination of the front, rear, left, and right sides of the terminal based on the body of the terminal in a horizontal direction with the ground. Information can be collected.

The screen direction recognition module of the controller 180 is obtained through the camera 121 of the A / V input unit 120, the wireless communication unit 110, and the sensing unit 120 in the direction obtaining step of the screen (S210). The direction of the playback screen is determined based on the tilt of the terminal detected in the tilt detection step (S200).

The determined screen direction may be set to landscape view or portrait view of the terminal according to the characteristics of the image, and the user may view the screen in the set landscape view or portrait view. When the screen orientation is recognized can be generated.

In the face direction obtaining step (S220), the screen direction determining module of the controller 180 may use face information, that is, a face or a face, photographed through a camera based on the direction information of the screen obtained in the direction obtaining step 210 of the screen. The direction of the face is extracted from the position of the face specific part.

Here, the extraction of the direction information on the face of the user, that is, the position of the face and a specific part of the face may be performed by, for example, identifying the short direction and the long direction of the face in the captured image of the user's face, It may include using a direction that is parallel with a specific position, or by using the direction of the two eyes in the image of the user's face to be photographed, using the direction and the direction that is horizontal to the identified eye. In addition, in the acquiring direction of the face (S220), the camera may recognize the image of the face at a predetermined time interval and may be selectively performed by a mode.

In the screen direction and face direction comparison step (S230), the screen direction determination module of the controller 180 is obtained in the screen direction information obtained in the direction obtaining step (S210) and the face direction obtaining step (S220). Compare facial orientations with each other.

For example, the direction or recognition of the face recognized from the image of the user's face taken in the screen direction and the face direction obtaining step (S220) generated in the horizontal or vertical view state of the screen set in the screen direction obtaining step (S210) By comparing the directions of certain parts of the face with each other, the slope between the screen direction and the direction of the face or the direction of the specific part of the face is parallel (0 degree), vertical (90 degree) or a certain range (for example, 0 to 45 degree). ) Can be compared.

In the determination step (S240), the screen direction determination module of the controller 180 compares the value compared in the screen direction and face direction comparison step (S230) with a predetermined threshold value.

In the screen orientation changing step 250, the screen orientation determination module of the controller 180 determines the screen orientation obtained when it is determined that the compared value is greater than the threshold value according to the determination of the determination step S240. Change it. For example, you can change the set landscape view to portrait view, or change the portrait view to landscape view.

In the screen orientation maintaining step (S260), the screen orientation determination module of the controller 180 determines the screen orientation obtained when it is determined that the compared value is smaller than the threshold value according to the determination of the determination step (S240). Keep it. For example, it is possible to maintain the set landscape view or portrait view.

For example, in the determination step 240 of FIG. 2, when the threshold value is assumed to be 45 degrees, the screen direction determination module of the controller 180 may determine the inclination of the screen direction and the direction of the face or the direction of a specific part of the face ( Or 45 degrees or more), it is determined that the screen direction and the direction of the face or the direction of a specific part of the face is in a range of inconsistency, and change the direction of the acquired screen, and the screen direction and the direction of the face or a specific part of the face. When the tilt (or angle) of the direction of the direction is 45 degrees or less, it may be determined that the screen direction and the direction of the face or a specific part of the face are in the same range to maintain the obtained screen direction as it is.

Accordingly, the present invention compares the orientation of the user's face looking at the screen or the direction of a specific part of the face with the screen direction without changing the horizontal view state or the screen portrait state of the terminal screen of the user device, and converts the screen view state. It can determine whether or not to do so and automatically switch the screen view state if necessary.

3 is an exemplary diagram illustrating a method of determining a screen direction according to a direction of a face through image recognition according to an exemplary embodiment disclosed herein.

3 illustrates a screen direction generated in a horizontal or vertical view state of the screen set in the screen direction obtaining step (S210) and a direction of a face recognized from an image of a user face photographed in the face direction information obtaining step (S220). Are compared with each other and the screen orientation can be determined (or corrected) to match the recognized face orientation.

3 (a) shows the screen direction and the short direction and the long direction of the face in the photographed image of the user's face, compares the specific position of the short direction of the face with the direction of the face to be horizontal, and the screen direction. When the directions of the face and the face do not coincide with each other, the screen direction may be automatically changed so that the screen direction coincides with the face direction.

Figure 3 (b) is to grasp the short direction and the long direction of the face in the screen direction and the image of the user's face, and compare the specific position and the horizontal direction of the short direction of the face with each other, the screen direction and the face If the directions of 일치 match, the screen direction can be maintained as it is.

4 is an exemplary view illustrating a method of determining a screen direction according to a direction of a face specific part through image recognition according to an exemplary embodiment disclosed herein.

4 illustrates a specific part of a face recognized from a screen direction generated in a horizontal or vertical view state of the screen set in the direction information obtaining step S210 and an image of a user face photographed in the face direction obtaining step S220. It shows that the directions of the screens can be compared with each other and the screen directions can be determined (or corrected) to match the direction of a specific part of the face.

Figure 4 (a) is to determine the position of the two eyes in the screen direction and the image of the user's face, and compare the position of the identified eyes and the horizontal direction, and the screen direction and the position of the two eyes When the horizontal directions do not coincide with each other, the screen direction may be automatically changed to coincide with the positions of the eyes and the horizontal direction.

Figure 4 (b) is to determine the position of the two eyes in the screen direction and the image of the user's face, and compare the position of the two eyes and the horizontal direction, and the screen direction and the position of the two eyes When the horizontal directions coincide with each other, the screen direction can be maintained as it is.

FIG. 5 is a flowchart illustrating a method of determining a screen orientation using image recognition through mode selection by a mobile terminal according to an exemplary embodiment disclosed herein.

As shown in FIG. 5, the method for determining the screen direction using image recognition through the mode selection by the mobile terminal 100 includes a tilt detection step (S300), a screen direction acquisition step (S305), a mode selection step (S310), Face direction acquisition step (S315), face direction acquisition verification step (S320), error display step (S325), screen direction and face direction comparison step (S330), determination step (S335), screen direction change step (S340) and screen The direction maintaining step S345 is included.

In the tilt detection step (S300), the sensing unit 140 uses a variety of sensors, such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor, to determine the inclination of the front, rear, left, and right sides of the terminal on the basis of the body of the terminal in a horizontal direction with the ground. Information can be collected.

The screen direction recognition module of the controller 180 is obtained through the camera 121 of the A / V input unit 120, the wireless communication unit 110, and the sensing unit 120 in the direction obtaining step of the screen (S305). The direction of the playback screen is determined based on the tilt of the terminal detected in the tilt detection step (S300).

After the control unit 180 obtains the direction information of the screen in the mode selection step (S310) (S305), the face direction information of the user, that is, the position direction information of the face and a specific part of the face is additionally obtained through image recognition. Whether to use it or not is displayed on the terminal screen so that the user can select it.

If the user wants to determine (or correct) the acquired direction information by using the face direction information, the user may additionally select an image recognition mode. Otherwise, the user may obtain direction information without information on the face direction information through image recognition. The mobile terminal 100 will operate only in step S305.

In the face direction obtaining step (S315), the screen direction determining module of the controller 180 acquires the face information acquired through the camera based on the direction of the screen obtained in the direction obtaining step (S305), that is, the face or the face. The facial direction is extracted from the location of the specific part.

In the face direction acquisition verification step (S320), the screen direction determination module of the controller 180 determines whether the face direction acquisition step (S315) is operating correctly, and when the face direction acquisition step is not properly performed, Error indication can be displayed. For example, the user may be informed that the user is in an error state by using a color change of the screen, flickering of the screen, voice, and sound (S325).

In the screen direction and face direction comparison step (S330), the screen direction determination module of the controller 180 acquires the direction of the screen when the face direction of the user is properly obtained in the face direction acquisition verification step (S320). The direction of the screen acquired in S305 and the face direction obtained in the face direction obtaining step S315 are compared with each other.

In the determining step S335, the screen direction determining module of the controller 180 compares and compares the screen direction and the value compared in the face direction comparing step S330 with a predetermined threshold value.

In the screen direction changing step S340, the screen direction determining module of the controller 180 changes the screen direction when it is determined that the compared value is greater than the threshold value according to the determination in step S335. do.

In the screen orientation maintaining step S345, the screen orientation determination module of the controller 180 maintains the screen orientation as it is determined that the compared value is smaller than the threshold value according to the determination of the determination step S335. do.

With reference to the drawings illustrated as above, the apparatus and method according to at least one embodiment disclosed herein has been described, but the present specification is not limited by the embodiments and drawings disclosed herein, the invention Of course, various modifications can be made by those skilled in the art within the scope of the technical idea.

In addition, according to one embodiment disclosed herein, the above-described method may be implemented as code that can be read by a processor in a medium in which a program is recorded. Examples of the medium that can be read by the processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc., and may be implemented in the form of a carrier wave (e.g., transmission over the Internet) .

The mobile terminal 100 described above is not limited to the configuration and method of the embodiments described above, the embodiments are a combination of all or part of each embodiment selectively so that various modifications can be made It may be configured.

The embodiments disclosed herein have been described with reference to the accompanying drawings.

Here, terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary meanings, but should be construed as meaning and concept consistent with the technical idea disclosed in the present specification.

Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only examples described in the present specification, and not all of the technical ideas disclosed in the present specification are described. Therefore, various modifications It is to be understood that equivalents and modifications are possible.

100: mobile terminal 110: wireless communication unit
120: A / V input unit 130: user input unit
140: sensing unit 150: output unit
160: memory 170: interface unit
180: control unit 190: power supply unit

Claims (20)

Sensing the tilt of the body;
Recognizing a face direction through a camera;
Determining a screen orientation according to the detected inclination of the body and the direction of the face; And
Displaying a screen in the determined direction;
Screen orientation determination method of a mobile terminal comprising a. The method of claim 1,
The detecting of the inclination of the body may include detecting an inclination degree based on a body that is in a horizontal direction with the ground using an acceleration sensor, a gyro sensor, a geomagnetic sensor, or any combination thereof. . The method of claim 1,
Recognizing the direction of the face is a screen direction determination method of the mobile terminal, characterized in that for recognizing the direction of the face or a specific portion of the face. The method of claim 3,
Recognizing the direction of the face is a screen direction determination method of the mobile terminal, characterized in that the direction of the face in the image of the face to grasp the direction of the unidirectional direction of the specific direction. The method of claim 3,
Recognizing the direction of the specific part of the face is to determine the position of the two eyes in the image of the face, the screen direction determination method of the mobile terminal, characterized in that using the direction that is horizontal to the identified position of the two eyes. The method of claim 1,
Recognizing the direction of the face is a screen direction determination method of a mobile terminal, characterized in that the camera recognizes the image of the face at a predetermined time interval. The method of claim 1,
Recognizing the direction of the face is a screen direction determination method of the mobile terminal, characterized in that it is selectively performed by the mode. The method of claim 1,
The determining of the direction of the screen may include: comparing the direction of the screen with the direction of the face recognized from the image of the user's face or the direction of a specific part of the face in the horizontal or vertical view of the set screen. How to determine the screen orientation of the terminal. The method of claim 8,
The determining of the direction of the screen may include changing the set horizontal view state to the vertical view state if the inclination between the screen direction and the direction of the face or a specific part of the face does not fall within a predetermined range according to the comparison result. The screen orientation determination method of the mobile terminal, characterized in that for changing the view state to the landscape view state. The method of claim 8,
The determining of the direction of the screen may include maintaining the set horizontal or vertical view state when the inclination with respect to each other between the screen direction and the direction of the face or a specific part of the face falls within a predetermined range according to the comparison result. Characterized in that the direction of the screen of the mobile terminal. Sensing unit for detecting the tilt of the body;
Through the camera to recognize the direction of the face,
A controller configured to determine a direction of a screen according to a slope of the body and a direction of a face; And
An output unit which displays a screen in the determined direction;
Screen orientation determination device of a mobile terminal, comprising a. The method of claim 11, wherein
The sensing unit for detecting the inclination of the body based on the body in the horizontal direction with respect to the ground to determine the degree of inclination using an acceleration sensor, a gyro sensor or any combination thereof. The method of claim 11, wherein
And a controller for recognizing the direction of the face recognizes a position of a face or a specific part of the face. The method of claim 13
The controller for recognizing the direction of the face detects the short direction and the long direction of the face in the image of the face, and uses a direction that is parallel to a specific position of the short direction of the face. . The method of claim 13
The control unit for recognizing the direction of the specific part of the face to determine the position of the two eyes in the image of the face, the screen orientation determination device of the mobile terminal, characterized in that using the direction that is horizontal to the position of the identified eye. The method of claim 11, wherein
The controller for recognizing the direction of the face is a screen direction determination apparatus of the mobile terminal, characterized in that the camera recognizes the image of the face at a predetermined time interval. The method of claim 11, wherein
The controller for recognizing the direction of the face is selectively performed by the mode of the screen direction determining apparatus of the mobile terminal. The method of claim 11, wherein
The controller for determining the orientation of the screen may compare the orientation of the screen and the orientation of a specific part of the face recognized from the photographed image of the user's face with the landscape or portrait view of the set screen. Device for determining the screen orientation of the mobile terminal. The method of claim 18, wherein
The controller which determines the orientation of the screen may change the set landscape view state to portrait view state if the inclination with respect to each other between the screen direction and the direction of the face or a specific part of the face does not fall within a predetermined range according to the comparison result. And changing the set portrait view state to landscape view state. The method of claim 18, wherein
The controller which determines the orientation of the screen maintains the set landscape view or portrait view when the inclination of each other between the screen direction and the direction of the face or a specific part of the face falls within a predetermined range according to the comparison result. An apparatus for determining the screen orientation of a mobile terminal.

Images