KR102143500B1 - Mobile terminal and control method thereof - Google Patents

Abstract

Disclosed are a mobile terminal and a control method thereof. A mobile terminal and a control method thereof according to the present invention include: a display; And at least one video is displayed on the display, and the at least one text is extracted from at least one of an image and a sound included in at least one area of the at least one video. It characterized in that it comprises a control unit to display on the display in association with the point. According to the present invention, it is possible to more conveniently manipulate the video by displaying the video and sound of the video as text.

Description

Mobile terminal and its control method {MOBILE TERMINAL AND CONTROL METHOD THEREOF}

The present invention relates to a mobile terminal, and more particularly, to a mobile terminal capable of more conveniently manipulating a video by displaying an image and sound of a video as text, and a control method thereof.

Terminals such as personal computers, laptops, mobile phones, etc. are multimedia devices with complex functions such as taking photos or videos, playing music or video files, receiving games, and broadcasting as functions are diversified. It is implemented in the form of (Multimedia player).

Terminal can be moved Depending on whether or not it can be divided into a mobile terminal (mobile terminal) and a stationary terminal (stationary terminal). Again, mobile terminals can be divided into handheld terminals and vehicle mount terminals depending on whether the user can directly carry them.

In order to support and increase the function of the terminal, it is considered to improve the structural part and/or the software part of the terminal.

The present invention relates to a mobile terminal capable of more conveniently manipulating a video by displaying an image and sound of a video as text, and a control method thereof.

A mobile terminal according to an embodiment of the present invention for realizing the above object includes a display; And at least one video is displayed on the display, and the at least one text is extracted from at least one of an image and a sound included in at least one area of the at least one video. It may include a control unit to display on the display in association with the point.

When obtaining a selection signal for at least one point of the at least one video, the control unit may extract the at least one text from an extraction region including the at least one point.

The controller may change at least one of the range and number of the extraction regions according to at least one of a direction and a distance of a touch operation on the display.

The extraction region includes a first extraction region including the at least one point, a second extraction region adjacent to the first extraction region, and a third extraction region adjacent to the second extraction region, and the control unit , As the distance of the touch operation increases, the extraction range of the at least one text may be increased in the order of the first to third extraction regions.

The control unit may display the range of the extraction area on a progress bar that displays the playback state of the video.

The control unit may display at least one icon for identifying the start point and the end point at the start point and end point of the extraction region.

The controller may display the at least one text in the order of the extracted points based on the progress bar of the moving image.

There are a plurality of extracted texts, and the control unit may differently display at least one of the positions, sizes, and thicknesses of the plurality of texts.

The control unit may determine at least one of a location, a size, and a thickness of the plurality of texts based on the frequency of the at least one text.

The control unit may extract the at least one text based on at least one of a sentence unit, a paragraph unit, and a word unit.

The controller may extract and display at least one image related to the at least one text from the video.

When the control unit obtains a selection signal for at least one of the at least one text and the at least one image, the control unit may change another display corresponding to the selection signal.

The control unit displays at least one of information on a person included in the image, voice information included in the sound, behavior information of a person included in the image, and information on the number of people included in the image as the text. can do.

In addition, a control method of a mobile terminal according to an embodiment of the present invention for realizing the above-described object includes the steps of displaying at least one video; Extracting at least one text from at least one of an image and a sound included in at least one area of the at least one video; And displaying the at least one text on the display in association with a point at which the at least one text is extracted from the at least one video.

The extracting comprises: obtaining a selection signal for at least one point of the at least one video, setting an extraction region including the at least one point, and extracting the at least one text It may include the step of extracting from the region.

The setting may further include changing at least one of the range and number of the extraction regions according to at least one of a direction and a distance of a touch operation on the display.

The method may further include displaying the range of the extraction area on a progress bar displaying a playback state of the video.

The method may further include displaying at least one icon for identifying the starting point and the ending point at the starting and ending points of the extraction region.

The number of extracted texts is plural, and the displaying may include differently displaying at least one of a position, a size, and a thickness of the plurality of texts.

The method may further include extracting and displaying at least one image related to the at least one text from the video.

The mobile terminal and the control method thereof according to the present invention have an effect of more convenient manipulation of the moving picture by displaying the image and sound of the moving picture as text.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.
2A is a front perspective view of a mobile terminal according to an embodiment of the present invention.
2B is a rear perspective view of a mobile terminal according to an embodiment of the present invention.
3 is a flowchart showing an operation process of the mobile terminal of FIG. 1.
4 to 7 are diagrams illustrating the operation of the mobile terminal of FIG. 3.
8 to 11 are diagrams illustrating operations of the mobile terminal of FIG. 3 according to another embodiment.
12 and 13 are diagrams illustrating operations of the mobile terminal of FIG. 3 according to another embodiment.
14 to 16 are diagrams illustrating operations of the mobile terminal of FIG. 3 according to another embodiment.
FIG. 17 is a diagram illustrating an operation of the mobile terminal of FIG. 3 according to another embodiment.
18 and 19 are diagrams illustrating operations of the mobile terminal of FIG. 3 according to another embodiment.
FIG. 20 is a diagram illustrating an operation of the mobile terminal of FIG. 3 according to another embodiment.
FIG. 21 is a diagram illustrating an operation of the mobile terminal of FIG. 3 according to another embodiment.

The above-described objects, features and advantages of the present invention will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, in the present invention, various modifications may be made and various embodiments may be provided. Hereinafter, specific embodiments will be illustrated in the drawings and described in detail. The same reference numerals throughout the specification refer to the same elements in principle. In addition, when it is determined that a detailed description of known functions or configurations related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present specification are only identification symbols for distinguishing one component from other components.

Hereinafter, a mobile terminal related to the present invention will be described in more detail with reference to the drawings. The suffixes "module" and "unit" for components used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves.

Mobile terminals described herein may include a mobile phone, a smart phone, a laptop computer, a terminal for digital broadcasting, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, and the like. However, it will be readily apparent to those skilled in the art that the configuration according to the embodiment described in the present specification may be applied to a fixed terminal such as a digital TV or a desktop computer, except when applicable only to a mobile terminal.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.

The mobile terminal 100 includes a wireless communication unit 110, an audio/video (A/V) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory unit 160, and An interface unit 170, a control unit 180, and a power supply unit 190 may be included. Since the components shown in FIG. 1 are not essential, a mobile terminal having more components or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more modules that enable wireless communication between the mobile terminal 100 and a wireless communication system or between the mobile terminal 100 and a network in which the mobile terminal 100 is located. For example, the wireless communication unit 110 may include a broadcast reception module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The broadcast receiving module 111 receives a broadcast signal and/or broadcast-related information from an external broadcast management server through a broadcast channel.

The broadcast channel may include a satellite channel and a terrestrial channel. The broadcast management server may mean a server that generates and transmits a broadcast signal and/or broadcast-related information, or a server that receives and transmits a previously-generated broadcast signal and/or broadcast-related information to a terminal. The broadcast signal may include not only a TV broadcast signal, a radio broadcast signal, and a data broadcast signal, but also a broadcast signal in the form of a TV broadcast signal or a radio broadcast signal and a data broadcast signal combined.

The broadcast related information may mean information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcasting related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 112.

The broadcast related information may exist in various forms. For example, it may exist in the form of an Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB) or an Electronic Service Guide (ESG) of Digital Video Broadcast Handheld (DVBH).

The broadcast receiving module 111 receives broadcast signals using various broadcast systems, and in particular, Digital Multimedia Broadcasting Terrestrial (DMBT), Digital Multimedia Broadcasting Satellite (DMBS), Media Forward Link Only (MediaFLO), Digital Video Broadcast Handheld (DVBH). ), ISDBT (Integrated Services Digital Broadcast Terrestrial), and other digital broadcasting systems can be used to receive digital broadcasting signals. Of course, the broadcast receiving module 111 may be configured to be suitable for not only the digital broadcast system described above, but also other broadcast systems that provide broadcast signals.

Broadcast signals and/or broadcast related information received through the broadcast reception module 111 may be stored in the memory unit 160.

The mobile communication module 112 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal may include a voice call signal, a video call signal, or various types of data according to transmission/reception of text/multimedia messages.

The wireless Internet module 113 refers to a module for wireless Internet access, and the wireless Internet module 113 may be built-in or external to the mobile terminal 100. As a wireless Internet technology, WLAN (Wireless LAN) (WiFi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), and the like may be used.

The short-range communication module 114 refers to a module for short-range communication. As short-range communication technologies, Bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), Ultra Wideband (UWB), ZigBee, and the like may be used.

The location information module 115 is a module for checking or obtaining a location of a mobile terminal. A representative example of the location information module is a Global Position System (GPS) module. According to the current technology, the GPS module 115 calculates information on a distance from three or more satellites of a point (object) and information on a time at which the distance information is measured, and then calculates the calculated distance information. By applying the trigonometric method to, it is possible to calculate three-dimensional location information according to latitude, longitude, and altitude for a point (object) per hour. Further, a method of calculating position and time information using three satellites and correcting an error of the calculated position and time information using another satellite is also used. The GPS module 115 continuously calculates the current position in real time and calculates speed information using the current position.

Referring to FIG. 1, an audio/video (A/V) input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes an image frame such as a still image or a video obtained by an image sensor in a video call mode or a photographing mode. The processed image frame may be displayed on the display unit 151.

The image frame processed by the camera 121 may be stored in the memory unit 160 or transmitted to the outside through the wireless communication unit 110. Two or more cameras 121 may be provided depending on the configuration aspect of the terminal.

The microphone 122 receives an external sound signal by a microphone in a call mode, a recording mode, a voice recognition mode, and the like and processes it as electrical voice data. In the case of a call mode, the processed voice data may be converted into a form capable of being transmitted to a mobile communication base station through the mobile communication module 112 and then output. Various noise removal algorithms may be implemented in the microphone 122 to remove noise generated in the process of receiving an external sound signal.

The user input unit 130 generates input data for the user to control the operation of the terminal. The user input unit 130 may be composed of a key pad, a dome switch, a touch pad (positive pressure/electrostatic), a jog wheel, a jog switch, and the like.

The sensing unit 140 detects the current state of the mobile terminal 100, such as an open/closed state of the mobile terminal 100, the position of the mobile terminal 100, the presence of user contact, the orientation of the mobile terminal, and acceleration/deceleration of the mobile terminal. Thus, a sensing signal for controlling the operation of the mobile terminal 100 is generated. For example, when the mobile terminal 100 is in the form of a slide phone, it is possible to sense whether the slide phone is opened or closed. In addition, a sensing function related to whether the power supply unit 190 supplies power and whether the interface unit 170 is coupled to an external device may be performed. Meanwhile, the sensing unit 140 may include a proximity sensor.

The output unit 150 is for generating output related to visual, auditory, or tactile sense, and includes a display unit 151, an audio output module 152, an alarm unit 153, and a haptic module 154. I can.

The display unit 151 displays and outputs information processed by the mobile terminal 100. For example, when the mobile terminal is in a call mode, a user interface (UI) or a graphical user interface (GUI) related to a call is displayed. When the mobile terminal 100 is in a video call mode or a photographing mode, a photographed or/and received image, UI, or GUI is displayed.

The display unit 151 is a liquid crystal display, a thin film transistor liquid crystal display, an organic light-emitting diode, a flexible display, and a 3D display. It may include at least one.

Some of these displays may be configured as a transparent type or a light-transmitting type so that the outside can be seen through it. This may be referred to as a transparent display, and a representative example of the transparent display is a transparent LCD. The rear structure of the display unit 151 may also be configured as a light transmission type structure. With this structure, the user can see an object located behind the terminal body through an area occupied by the display unit 151 of the terminal body.

Two or more display units 151 may exist depending on the implementation form of the mobile terminal 100. For example, in the mobile terminal 100, a plurality of display units may be spaced apart or integrally disposed on one surface, or may be disposed on different surfaces.

When the display unit 151 and a sensor (hereinafter referred to as'touch sensor') for detecting a touch operation form a mutual layer structure (hereinafter, abbreviated as'touch screen'), the display unit 151 is an output device In addition, it can be used as an input device. The touch sensor may have a shape such as, for example, a touch film, a touch sheet, and a touch pad.

The touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 151 or a capacitance generated at a specific portion of the display unit 151 into an electrical input signal. The touch sensor may be configured to detect not only a touched position and area, but also a pressure at the time of touch.

When there is a touch input to the touch sensor, a signal(s) corresponding thereto is transmitted to the touch controller. The touch controller processes the signal(s) and then transmits the corresponding data to the controller 180. Accordingly, the controller 180 can know which area of the display unit 151 has been touched.

Referring to FIG. 1, a proximity sensor may be disposed in an inner area of a mobile terminal surrounded by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing in the vicinity using the force of an electromagnetic field or infrared rays without mechanical contact. Proximity sensors have a longer lifespan and higher utilization than contact sensors.

Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, an infrared proximity sensor, and the like.

When the touch screen is capacitive, it is configured to detect the proximity of the pointer by a change in an electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, the action of allowing the pointer to come close without contacting the touch screen to recognize that the pointer is located on the touch screen is referred to as "proximity touch", and the touch The act of actually touching the pointer on the screen is referred to as "contact touch". A position at which a proximity touch is performed with a pointer on the touch screen means a position at which the pointer vertically corresponds to the touch screen when the pointer is touched.

The proximity sensor detects a proximity touch and a proximity touch pattern (eg, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch movement state, etc.). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

The sound output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory unit 160 in a call signal reception, a call mode or a recording mode, a voice recognition mode, and a broadcast reception mode. The sound output module 152 outputs sound signals related to functions (eg, a call signal reception sound, a message reception sound, etc.) performed in the mobile terminal 100. The sound output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying the occurrence of an event in the mobile terminal 100. Examples of events occurring in the mobile terminal include call signal reception, message reception, key signal input, and touch input. The alarm unit 153 may output a signal for notifying the occurrence of an event in a form other than a video signal or an audio signal, for example, by vibration. The video signal or audio signal may also be output through the display unit 151 or the audio output module 152.

The haptic module 154 generates various tactile effects that a user can feel. A typical example of the tactile effect generated by the haptic module 154 is vibration. The intensity and pattern of the vibration generated by the haptic module 154 can be controlled. For example, different vibrations may be synthesized and output or may be sequentially output.

In addition to vibration, the haptic module 154 has an effect of stimulation by an arrangement of pins that move vertically with respect to the contact skin surface, an effect by stimulation through the injection force or suction force of air through the injection or inlet, and the stimulation through the skin surface. A variety of tactile effects can be generated, such as an effect by a stimulus through contact with an electrode, an effect by a stimulus using an electrostatic force, an effect by reproducing a feeling of cooling and warming using an endothermic or heat generating element.

The haptic module 154 may not only transmit a tactile effect through direct contact, but may also be implemented to feel the tactile effect through muscle sensations such as a user's finger or arm. Two or more haptic modules 154 may be provided depending on the configuration of the portable terminal 100.

The memory unit 160 may store a program for the operation of the controller 180 and may temporarily store input/output data (eg, a phone book, a message, a still image, a video, etc.). The memory unit 160 may store data regarding vibrations and sounds of various patterns output when a touch input on the touch screen is performed.

The memory unit 160 includes a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), At least one of RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (ReadOnly Memory, ROM), EEPROM (Electrically Erasable Programmable ReadOnly Memory), PROM (Programmable ReadOnly Memory) magnetic memory, magnetic disk, optical disk It may include a storage medium of the type. The mobile terminal 100 may operate in connection with a web storage that performs a storage function of the memory unit 160 over the Internet.

The interface unit 170 serves as a passage for all external devices connected to the mobile terminal 100. The interface unit 170 receives data from an external device or receives power and transmits it to each component inside the mobile terminal 100 or transmits data inside the mobile terminal 100 to an external device. For example, a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio input/output (I/O) port, A video input/output (I/O) port, an earphone port, and the like may be included in the interface unit 170.

The identification module is a chip that stores various types of information for authenticating the right to use the mobile terminal 100, and includes a user identification module (UIM), a subscriber identification module (SIM), and a universal user authentication module ( Universal Subscriber Identity Module, USIM), etc. may be included. A device equipped with an identification module (hereinafter referred to as'identification device') may be manufactured in the form of a smart card. Accordingly, the identification device may be connected to the terminal 100 through the port.

When the mobile terminal 100 is connected to an external cradle, the interface unit becomes a path through which power from the cradle is supplied to the mobile terminal 100, or various command signals input from the cradle by a user are moved. It can be a passage to the terminal. Various command signals or the power input from the cradle may be operated as signals for recognizing that the mobile terminal is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the mobile terminal. For example, it performs related control and processing for voice calls, data communication, and video calls. The controller 180 may also include a multimedia module 181 for playing multimedia. The multimedia module 181 may be implemented in the controller 180 or may be implemented separately from the controller 180.

The controller 180 may perform a pattern recognition process capable of recognizing a handwriting input or a drawing input performed on the touch screen as characters and images, respectively.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for the operation of each component.

Various embodiments described herein may be implemented in a recording medium that can be read by a computer or a similar device using, for example, software, hardware, or a combination thereof.

According to hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented using at least one of processors, controllers, microcontrollers, microprocessors, and electrical units for performing a function. 180).

According to the software implementation, embodiments such as procedures or functions may be implemented together with separate software modules that perform at least one function or operation. The software code can be implemented by a software application written in an appropriate programming language. In addition, the software code may be stored in the memory unit 160 and executed by the controller 180.

2A is a front perspective view of an example of a mobile terminal or a portable terminal related to the present invention.

The disclosed portable terminal 100 has a bar-shaped terminal body. However, the present invention is not limited thereto, and can be applied to various structures such as a slide type, a folder type, a swing type, and a swivel type in which two or more bodies are coupled so as to be relatively movable.

The body includes a case (casing, housing, cover, etc.) forming the exterior. In this embodiment, the case may be divided into a front case 101 and a rear case 102. Various electronic components are embedded in the space formed between the front case 101 and the rear case 102. At least one intermediate case may be additionally disposed between the front case 101 and the rear case 102.

The cases may be formed by injection of synthetic resin or may be formed of a metal material such as stainless steel (STS) or titanium (Ti).

A display unit 151, an audio output unit 152, a camera 121, a user input unit 130/131, 132, a microphone 122, an interface 170, etc. may be disposed on the terminal body, mainly the front case 101. have.

The display unit 151 occupies most of the main surface of the front case 101. An audio output unit 152 and a camera 121 are disposed in an area adjacent to one end of the display unit 151, and a user input unit 131 and a microphone 122 are disposed in an area adjacent to the other end. The user input unit 132 and the interface 170 are disposed on side surfaces of the front case 101 and the rear case 102.

The user input unit 130 is manipulated to receive a command for controlling the operation of the portable terminal 100 and may include a plurality of manipulation units 131 and 132.

The manipulation units 131 and 132 may be collectively referred to as a manipulating portion, and any method may be employed as long as the user operates while having a tactile feeling.

Contents input by the manipulation units 131 and 132 may be set in various ways. For example, the first manipulation unit 131 receives commands such as start, end, scroll, etc., and the second manipulation unit 132 adjusts the size of sound output from the sound output unit 152 or the display unit 151 A command such as switching to the touch recognition mode of) can be input.

2B is a rear perspective view of the portable terminal shown in FIG. 2A.

Referring to FIG. 2B, a camera 121 ′ may be additionally mounted on the rear surface of the terminal body, that is, the rear case 102. The camera 121 ′ has a photographing direction substantially opposite to that of the camera 121 (see FIG. 2A ), and may be a camera having different pixels from the camera 121.

For example, the camera 121 has a low pixel so that it is not unreasonable to photograph the user's face and transmit it to the other party in the case of a video call, and the camera 121 ′ photographs a general subject and does not immediately transmit it. It is desirable to have a high number of pixels. The cameras 121 and 121 ′ may be installed on the terminal body to enable rotation or popup.

A flash 123 and a mirror 124 are additionally disposed adjacent to the camera 121 ′. The flash 123 illuminates light toward the subject when the subject is photographed by the camera 121 ′. The mirror 124 enables the user's own face to be projected when the user wants to take a picture (self-photographing) using the camera 121'.

An audio output unit 152' may be additionally disposed on the rear surface of the terminal body. The sound output unit 152 ′ may implement a stereo function together with the sound output unit 152 (refer to FIG. 2A ), and may be used to implement a speaker phone mode during a call.

An antenna 124 for receiving broadcast signals may be additionally disposed on the side of the terminal body in addition to an antenna for a call. An antenna 124 forming a part of the broadcast receiving module 111 (refer to FIG. 1) may be installed to be retractable from the terminal body.

A power supply unit 190 for supplying power to the mobile terminal 100 is mounted on the terminal body. The power supply unit 190 may be built in the terminal body, or may be configured to be directly detachable from the outside of the terminal body.

A touch pad 135 for sensing a touch may be additionally mounted on the rear case 102. Like the display unit 151, the touch pad 135 may be configured as a light transmitting type. In this case, if the display unit 151 is configured to output visual information from both sides, the visual information can also be recognized through the touch pad 135. All of the information output on both sides may be controlled by the touch pad 135. Alternatively, a display may be additionally mounted on the touch pad 135, and a touch screen may also be disposed on the rear case 102.

The touch pad 135 operates in association with the display unit 151 of the front case 101. The touch pad 135 may be disposed parallel to the rear of the display unit 151. The touch pad 135 may have the same or smaller size as the display unit 151.

3 is a flowchart showing an operation process of the mobile terminal of FIG. 1.

As shown, the mobile terminal 100 according to an embodiment of the present invention may extract text from voice and/or image included in video content and display it on the display 151.

The controller 180 may perform an operation S10 of displaying the video content.

The video content may be data including an image that is a continuous motion. The video content may include sound along with an image. For example, there may be cases in which an image and sound are included in the entire video, only an image is included, or sound is not included in at least some sections.

The controller 180 may perform an operation (S20) of receiving a selection of a specific point or region of the video content.

The user can select a specific point or area of the video content. For example, this means that it is possible to select a specific point or area of the video content being played or the video content in the preparation stage for playback.

Selection of a specific point or region of the video content may be made through an operation of touching a progress bar displaying the playback time or status of the video content with a finger or the like. Selection of a specific point or area of the moving picture content may be made through a touch operation of touching a specific point of the progress bar and then dragging in a certain direction. Selection of a specific point or region of the video content may be made according to a control signal of the controller 180 based on a specific point of view and/or a reference.

The controller 180 may perform an operation S30 of analyzing an image and/or sound included in a selected specific point or region and an operation S40 of extracting text from the extracted image and/or sound.

As described above, the video content is data including an image, and the image may include sound. The controller 180 may analyze an image and/or sound included in the video content. The controller 180 may extract text from the extracted image and/or sound.

Extracting text from an image may mean the following. For example, an image may include a plurality of people. The controller 180 may analyze how many people are included at a specific point in time of the image. The controller 180 may extract text from the analyzed number of people. For example, this means that you can create the text “three people”. Also, the controller 180 may analyze the name of a person included in a specific viewpoint of the image. That is, based on the facial recognition technology, it is possible to analyze who the person included in the specific point of view of the image is by comparing the face of the person included in the image with the face of the person previously stored. When the name of the person is analyzed, the analyzed person's name can be displayed as text. For example, it means you can create the text "John".

Extracting a texture from sound can mean the following. For example, the voice “Happy birthday to you” may be included in a specific viewpoint or region of the image. The controller 180 may analyze the voice included in the image. , Based on the analysis, text can be extracted from the speech.

The controller 180 may perform an operation S50 of displaying the extracted text. For example, the controller 180 may display the extracted text “three people”, “John”, “Happy birthday to you”, and the like on the display 151. The point at which the text is displayed may be a point at which an event that causes the text to be generated occurs. For example, it means that text can be displayed at the point where the text is generated based on the progress bar of the video.

The controller 180 may perform an operation (S60) of editing video content based on the displayed text.

The controller 180 may edit the video content. Editing of the video content may be performed based on the displayed text. For example, in a 10-minute video content, the portion where John is photographed may be between the 3 minute point and the 5 minute point of the video. The controller 180 may selectively store only a portion between the 3 minute point and the 5 minute point where John is photographed according to a control signal of the controller 180 and/or a user's selection.

4 to 7 are diagrams illustrating the operation of the mobile terminal of FIG. 3.

As shown in these drawings, the controller 180 of the mobile terminal 100 according to an embodiment of the present invention may display text based on an image and/or sound included in a video.

As illustrated in (a) of FIG. 4, a moving image C may be displayed on the display 151. Various images and/or sounds may be included in the video (C).

The controller 180 may display a progress bar PB corresponding to the video C on the display 151. For example, this means that the progress bar PB corresponding to the entire length of the video C can be displayed.

The controller 180 may display at least one selection button B. The selection button B may be a button capable of receiving a user's input for a source to be analyzed. For example, the selection button (B) is a first button (B1) for analyzing sound, a second button (B2) for analyzing behavior, and a third button for analyzing people. It may include a button (B3).

When the first button B1 is selected, the controller 180 may analyze the sound included in the video C. For example, this means that it is possible to analyze a voice captured in a video at a specific time point or section.

When the second button B2 is selected, the controller 180 may analyze the movement of the object included in the video C. For example, it means that it is possible to analyze the fact that a vehicle passes, a person is photographed, or a building has been photographed at a specific point in time or section.

When the third button B3 is selected, the controller 180 may analyze a person included in the video C. For example, this means that analysis to identify a person photographed at a specific point in time or section may be possible. Information about the person's name, etc. can be extracted through facial recognition of the person photographed.

As shown in (b) of FIG. 4, the user may select a second button B2 through which a finger F or the like can be used to analyze the motion of the object contained in the video C.

As illustrated in FIG. 5A, when the second button B2 is selected, the controller 180 may analyze the video C and display a specific event as text. It means that the extracted text (AW) for a specific movement can be displayed on the progress bar (PB). For example, the extracted text AW may be a description of what movements of people photographed in the video C, such as'walk' and'watch', are.

The controller 180 may display the extracted text AW at the time when the operation is performed. For example, it means that the extracted text (AW) can be displayed on the progress bar (PB), but the extracted text (AW) can be displayed at the point where the operation was performed.

As shown in (b) of FIG. 5, the user may select at least one of the extracted texts AW displayed with a finger F or the like. For example, you can select the extracted text (AW) called'talk'.

As shown in (a) of FIG. 6, the controller 180 may display an extraction area AB related to the selected specific extracted text AW.

The extraction area AB may be an area in which contents related to the extracted text AW are displayed. For example, if the extracted text (AW) is'talk', the controller 180 displays the section in which the people photographed in the video (C) perform the'talk' operation on the progress bar (PB). Means you can. The controller 180 may display the color, shade, size, etc. of the extraction area AB differently from other parts of the progress bar PB.

As shown in (b) of FIG. 6, the user can select the extraction area AB using a finger F or the like. The user's selection operation may be a touch operation TD of moving in a specific direction after touching the extraction area AB. For example, it may be a touch operation in which a first drag TD1 upward by touching the extraction area AB or a second drag TD2 downward is performed. The controller 180 may perform different operations based on the direction and size of the touch operation.

As shown in (a) of FIG. 7, the user may perform a first drag TD1 or a second drag TD2 in different directions by touching the extraction area AB. The controller 180 may perform different operations according to the direction of the drag. For example, when the first drag TD1 is input, the storage function for the selection area AB is performed, and when the second drag TD2 is input, the deletion function for the selection area AB can be performed. it means.

As shown in FIG. 7B, the controller 180 may execute different functions for the first and second drags TD1 and TD2 in the same direction. The first and second drags TD1 and TD2 may have the same direction, but may have different drag lengths. For example, the first drag TD1 may be dragged by the first length TR1 and the second drag TD2 may be dragged by the second length TR2. The controller 180 may execute different functions when a preset criterion is not met or overflows. For example, an extraction area AB related to the extracted text AW'talk' is displayed up to the first length TR1, and when the second length TR2 is reached, more areas corresponding to the extracted text AW are displayed. It means that you can display it.

8 to 10 are diagrams illustrating operations of the mobile terminal of FIG. 3 according to another embodiment.

As shown in these drawings, the control unit 180 of the mobile terminal 100 according to another embodiment of the present invention displays text included in a video and determines a range of text displayed in response to a user's input. Can be adjusted.

As shown in (a) of FIG. 8, the user can touch a specific point of the progress bar PB.

The controller 180 may display a first image C1 related to a specific point touched in the video content. For example, it means that a thumbnail image of a portion of the video corresponding to the point touched by the user in the progress bar (PB) can be displayed.

The controller 180 may display the first text EW1 related to the touched specific point. For example, the voice included in the video of the extraction area AB related to the touch point of the progress bar PB may be'Happy Birth day'. The controller 180 may analyze the voice included in the video and display the voice as text.

The controller 180 may differently display a text close to a specific point touched among the first text EW1 of the extraction area AB. For example, when'Birth' is close to a point touched by the finger F or the like, unlike other texts,'Birth' may be displayed larger and/or thicker.

As shown in FIG. 8B, the user may perform a touch operation of dragging the touched finger F. For example, an operation of dragging the first touched progress bar PB downward by the first distance DR1 may be performed.

As described above, the controller 180 can display the extraction area AB when the progress bar PB is touched. The extraction area AB may be expressed differently from other parts of the progress bar PB.

The controller 180 may display an area symbol FL capable of clearly indicating the start and end of the extraction area AB. For example, it means that at least one of a color and a shape can be displayed at the start point and end point of the extraction area AB. When the area symbol FL is displayed, the user can clearly recognize the start and end points of the area related to the displayed first text EW1.

The controller 180 may perform an operation corresponding to a distance dragged by the user by the first distance DR1 using the finger F. For example, when a drag operation corresponding to the first distance DR1 is input, the controller 180 may display an image adjacent to the extraction area AB. That is, it means that in addition to the first image C1 corresponding to the extraction area AB, the second and third images C2 and C3 before and after the extraction area AB can be further displayed.

As shown in (a) of FIG. 9, using a finger F or the like, the user may perform a drag operation by a second distance DR2 that exceeds the first distance DR1. When the drag distance increases, the controller 180 may perform another operation.

When the drag distance using the finger F or the like is the first distance DR1, the controller 180 displays the first to third images C1 to C3, and the drag distance increases to the second distance DR2. After that, the fourth and fifth images C4 and C5 can be further displayed.

The controller 180 may increase the range of the extracted text as the display of the image increases. For example, when dragged by the first distance DR1,'Happy Birthday', the voice included in the first extraction area AB1, is extracted and displayed as text, but when dragged the second distance DR1, the first Voices'Mom' and'I love you' included in the second and third extraction areas AB2 and AB3 adjacent to the extraction area AB1 may be further displayed as text. In other words, it means that the range for extracting the text can be increased in proportion to the drag distance.

As shown in (b) of FIG. 9, the user can select a specific image among the representative images AC with a finger F or the like. For example, it means that the first image C1 can be selected.

The controller 180 may change at least one of a size, color, and thickness of text corresponding to the selected image. For example, if the first image C1 is selected, it means that'Birth', which is a text corresponding to the voice at the viewpoint of the first image C1, can be displayed in large and bold, unlike other texts.

As illustrated in FIG. 10, the user can select the first extraction area AB1 by garting. For example, it means that an operation of touching the first extraction area AB1 with a finger F or the like can be performed. When the first extraction area AB1 is selected, the controller 180 may display the selection menu SM. The selection menu SM may be a hidden menu that is not normally displayed but is displayed by a specific operation of the user.

The user may select the first extraction area AB1 and then perform a touch operation of dragging to a specific menu among the selection menu SM. The controller 180 may perform a function related to the selection menu SM corresponding to the point where the user's touch operation is terminated.

As shown in FIG. 11, the drag distance and the range in which the text is extracted and displayed may have a proportional relationship. The horizontal axis of the displayed graph may be a drag distance, and the vertical axis may be a range of text to be extracted and displayed.

A user who touches a specific point can increase the drag distance as shown in D1 and D2. The controller 180 may initially increase the range to extract text to the range a, the range b when the drag distance becomes D1, and the range c when the drag distance becomes D2. That is, it means that the range from which the text is to be extracted can be gradually increased according to the drag distance.

12 and 13 are diagrams illustrating operations of the mobile terminal of FIG. 3 according to another embodiment.

As shown in these drawings, the controller 180 of the mobile terminal 100 according to another embodiment of the present invention selects a progress bar (PB) to select a specific part of the video, The operation can be performed.

As shown in (a) of FIG. 12, the user can touch the progress bar PB. The controller 180 may display a representative image AC corresponding to the touched point of the progress bar PB. When the user drags a finger F or the like and moves on the progress bar PB, the controller 180 may display another representative image AC corresponding to the moved point.

As shown in (b) of FIG. 12, a user who has moved on the progress bar PB by using a finger F or the like can stop the horizontal movement of the finger F at a specific position.

As shown in (c) of FIG. 12, the user may perform an operation of dragging a finger F or the like downward from a specific position. When the drag starts downward, the controller 180 may display an area related to the representative image AC of the corresponding location as the extraction area AB. First and second area symbols FL1 and FL2 may be displayed at the start and end points of the extraction area AB.

As shown in FIG. 13A, the controller 180 may perform a control operation according to an increase in the drag distance. The user may continuously drag the finger F or the like in a specific direction. For example, this means that the drag distance can be gradually increased, such as the first distance DR1, the second distance DR2, and the third distance DR3.

The controller 180 may increase the extraction area AB according to an increase in the drag distance. For example, when the drag distance is the first distance DR1, the first extraction area AB1 is displayed, and when the drag distance is increased to the second distance DR2, the second and third extraction areas AB2, When AB3) is displayed and the drag distance is increased to the third distance DR3, it means that the fourth and fifth extraction regions AB4 and AB5 can be displayed.

The controller 180 may display first to sixth area symbols FL1 to FL6 to clearly indicate the boundary of each extraction area AB.

Each extraction area AB may have different properties of an image and/or sound included in the video. For example, each extraction area AB may be an independent sentence. That is, it means that the first extraction area AB1 may be one sentence, and the second and third extraction areas AB2 and AB3 may be another sentence. The controller 180 may display the extracted sentences in the vicinity of each extraction area AB.

Each extraction area AB may be an area in which a different event has occurred. For example, this means that each extraction area AB can be classified based on a point where a change has occurred in the image, such as a new person appearing in the video, a person disappearing, or a vehicle passing by.

As shown in FIG. 13B, a user may perform a touch operation of selecting at least one of the first to sixth area symbols FL1 to FL6 with a finger F or the like. When at least one of the first to sixth area symbols FL1 to FL6 is selected, the controller 180 may display a function related to the selected area symbol. For example, when the second area symbol FL2 is selected, the controller 180 deletes, stores, and shares the first extraction area AB1 corresponding to the second area symbol FL2. ) Can be displayed.

14 to 16 are diagrams illustrating operations of the mobile terminal of FIG. 3 according to another embodiment.

As shown in these drawings, the controller 180 of the mobile terminal 100 according to an embodiment of the present invention may extract text included in a video based on a certain criterion.

As shown in FIG. 14A, the controller 180 may display a progress bar PB corresponding to the selected video content on the display 151. The controller 180 may display a reference unit button CR for selecting a criterion for extracting text. For example, it means that it is possible to select which unit of text to be extracted and displayed among sentence units, paragraph units, and word units. The user can select a specific one of the reference unit buttons CR. For example, you can select a button to extract text by word. Hereinafter, a case of extracting text in a word unit will be described, thereby replacing a case of selecting a different criterion.

As shown in FIG. 14B, when a word unit is selected, the controller 180 may extract and display text in word units based on a specific point of the selected video. For example, the user may select a specific point of the progress bar PB with a finger F or the like. When a specific point is selected, the controller 180 may determine which sound is included based on the corresponding point of the video. For example, the sound “happy” may be included within a certain range based on the selected point. The controller 180 may extract the sound'happy' and display it as the first text EW1. The controller 180 may display a portion corresponding to the sound'happy' as the first extraction area AB1. The controller 180 may display the first representative image AC1 of the first extraction area AB1.

As shown in (a) of FIG. 15, the user can drag the finger F down by the first distance DR1.

As shown in (b) of FIG. 15, when the user drags the finger F downward by the first distance DR1, the controller 180 moves the text to another area near the first extraction area AB1. The range of extraction can be increased. For example, this means that the text extraction range can be widened to the second and third extraction regions AB2 and AB3 in front of and behind the first extraction region AB1. The controller 180 may also extract text in the second and third extraction areas AB2 and AB3 according to the same criteria as the first extraction area AB1. The controller 180 may display the extracted text as second and third texts EW2 and EW3.

The controller 180 may display the second and third representative images AC2 and AC3 corresponding to the second and third extraction regions AB2 and AB3. That is, it means that a thumbnail image can be extracted and displayed from the videos in the second and third extraction areas AB2 and AB3, respectively.

When the user drags the finger F or the like further by the second distance DR2, the range for extracting the text may increase beyond the second and third extraction areas AB2 and AB3.

As shown in FIG. 16, the user may select at least one of the displayed representative images AC1 to AC3. For example, a third representative image AC3 may be selected. The controller 180 may display the third text EW3 corresponding to the selected third image AC3 differently from other texts. For example, this means that the font of the third text EW3 can be displayed in a large or bold font. That is, when one of the image and the text is selected, it means that the other display corresponding to the selected one can be changed.

FIG. 17 is a diagram illustrating an operation of the mobile terminal of FIG. 3 according to another embodiment.

As shown, the controller 180 of the mobile terminal 100 according to another embodiment of the present invention can extract and display text even when only an image exists in a video.

As shown in (a) of FIG. 17, the controller 180 extracts the first text EW1 and the first image AC1 from the first extraction area AB1 centered on a specific point selected by the user. Can be displayed.

As shown in (b) of FIG. 17, the extraction area may be expanded by manipulation by the user and/or the controller 180. For example, it means that it can be extended to the second and/or third extraction regions AB2 and AB3 adjacent to the first extraction region AB1.

The first and second extraction regions AB1 and AB2 may include both an image and a sound. The controller 180 may extract the first and second texts EW1 and EW2 based on sound included in the video. The controller 180 may display the first and second representative images AC1 and AC2 based on the image included in the moving image.

Only an image may be included in the third extraction area AB3. The controller 180 may analyze the image itself in the third extraction area AB3 including only the image. That is, it means that text can be extracted through analysis of characteristic situations included in the image. For example, the controller 180 may extract and display the third text EW3 called'party' through analysis of the image included in the third extraction area AB3.

18 and 19 are diagrams illustrating operations of the mobile terminal of FIG. 3 according to another embodiment.

As shown in these drawings, the controller 180 of the mobile terminal 100 according to another embodiment of the present invention may display information of a person photographed through analysis of a video in various ways.

As illustrated in (a) of FIG. 18, the controller 180 may analyze a person included in a video included in the first to third extraction areas AB1 to AB3. For example, it means that facial recognition information extracted from a person included in a video can be compared with pre-stored information to extract the name of a person in the video. Through analysis of a person, the controller 180 may display text information on which person has been photographed in a specific area. For example, John and Suji were photographed in the third extraction region (AB3), John was photographed in the first extraction region (AB1), and John, Tom, and Suji were photographed in the second extraction region (AB2). It means that it can be displayed as name text (PN1).

As shown in FIG. 18B, the controller 180 may analyze how many people are included in the video included in the first to third extraction areas AB1 to AB3. For example, the controller 180 may determine that the first extraction area AB1 includes two people through analysis of the image. The controller 180 that has determined the number of people may display the number of people corresponding to the first to third extraction areas AB1 to AB3 as the number of people text PN2.

As shown in (a) of FIG. 19, the controller 180 may display identification information (AP) of a person included in the video. The person identification information (AP) may be the name of a person included in the video.

The controller 180 may display the identification information AP on the part where the person is photographed. That is, it means that the identification information (AP) can be displayed on the video based on the point where the person was first photographed. Therefore, John can easily know with the identification information (AP) indicated that the film was taken earlier than Tom.

The controller 180 may display the identification information AP based on importance such as a frequency of a person photographed and a photographing time. That is, a person with a high frequency of being photographed may display at least one of a text size, a thickness, and a font differently than other people. For example, in the video, you can feel intuition through the identification information (AP) indicating that John was taken relatively principally.

As shown in (b) of FIG. 19, the user can select a specific person with a finger (F) or the like. For example, you can choose John. When a specific person is selected, the controller 180 may display the photographing area SR of the person selected from the video.

FIG. 20 is a diagram illustrating an operation of the mobile terminal of FIG. 3 according to another embodiment.

As shown, the controller 180 of the mobile terminal 100 according to another embodiment of the present invention may set a region through various analysis of images and/or sounds of a moving picture.

The controller 180 may obtain analysis data EG through analysis of an image and/or sound of a moving picture. For example, the analysis data EG may be a change in sound included in a video. That is, it may be data analyzed over time as to what kind of change has occurred in the loudness of the sound, etc. For example, the analysis data EG may be a change in an image included in a video. That is, it may be data obtained by analyzing a change in a person included in the image, a shake in the image, a change in color or brightness of the image, a change in weather in the image, and a change in a place in the image.

The controller 180 may divide the video into at least two regions through analysis of an image and/or sound of the video. For example, the first area A1 may be an area in which scenery is photographed, and the second area A2 may be an area in which one person is photographed. The controller 180 may differentiate and display the first area A1 and the second area A2 based on the difference between the image of the scenery and the person. Meanwhile, the third area A3 may be an area in which a plurality of people are photographed and a relatively loud voice is photographed. The controller 180 may differentiate and display the second area A2 and the third area A3 based on the number of people and/or sound.

FIG. 21 is a diagram illustrating an operation of the mobile terminal of FIG. 3 according to another embodiment.

As shown, the control unit 180 of the mobile terminal 100 according to the present invention can distinguish and display an audio and an image of a single video.

The controller 180 may display the first and second progress bars PB1 and PB2. The first progress bar PB1 may be a part corresponding to the audio of the moving picture, and the second progress bar PB2 may be a part corresponding to the image of the moving picture. The controller 180 includes at least one of the first and second progress bars (PB1, PB2) with text (EW) extracted from the voice of the video, a representative image (AC) and/or the identification information (AP) extracted from the video of the video Can be displayed in a specific location.

The video may be edited according to a control signal from the controller 180 and/or a user's selection. For example, this means that it is possible to selectively merge only the part containing John and the part containing Love.

It is apparent to those of ordinary skill in the art that the present invention is not limited to the described embodiments, and that various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, it should be said that such modifications or variations belong to the claims of the present invention.

100: mobile terminal 110: wireless communication unit
140: sensing unit 151: display unit
180: control unit

Claims (20)

display; And
At least one moving picture and a progress bar related to the moving picture are displayed on the display, and at least from the moving picture in an extraction area including the one point in response to a touch input to a point of the progress bar A control unit for extracting one text and displaying the extracted text to be associated with the extraction area;
Including,
The control unit changes at least one of the range or number of the extraction regions based on a drag length of the touch input, and extracts at least one text from the video in the changed extraction region. terminal. delete delete The method of claim 1,
The extraction area,
A first extraction region including the at least one point, a second extraction region adjacent to the first extraction region, and a third extraction region adjacent to the second extraction region,
The control unit,
And increasing the extraction range of the at least one text in the order of the first to third extraction regions as the distance of the touch operation increases. The method of claim 1,
The control unit,
And displaying the range of the extraction area on the progress bar displaying the playback state of the video. The method of claim 5,
The control unit,
And displaying at least one icon for identifying the starting and ending points of the extraction region. The method of claim 1,
The control unit,
And displaying the at least one text in the order of the extracted points based on a progress bar of the moving picture. The method of claim 1,
The extracted text is plural,
The control unit,
And displaying at least one of a position, a size, and a thickness of the plurality of texts differently. The method of claim 8,
The control unit,
And determining at least one of a position, a size, and a thickness of the plurality of texts based on the frequency of the at least one text. The method of claim 1,
The control unit,
And extracting the at least one text based on at least one of a sentence unit, a paragraph unit, and a word unit. The method of claim 1,
The control unit,
And extracting and displaying at least one image related to the at least one text from the video. The method of claim 11,
The control unit,
And when a selection signal for at least one of the at least one text and the at least one image is obtained, another display corresponding to the selection signal is changed. The method of claim 1,
The control unit,
Displaying at least one of information on a person included in the image, voice information included in the sound, behavior information of the person included in the image, and information on the number of people included in the image as the text Mobile terminal. delete delete delete delete delete delete delete

Images