WO2019233318A1

WO2019233318A1 - Content identification method and device, and mobile terminal

Info

Publication number: WO2019233318A1
Application number: PCT/CN2019/088874
Authority: WO
Inventors: 段丽霞
Original assignee: Oppo广东移动通信有限公司
Priority date: 2018-06-08
Filing date: 2019-05-28
Publication date: 2019-12-12
Also published as: CN108958576A; CN108958576B

Abstract

Embodiments of the present application relate to the technical field of mobile terminals. Disclosed are a content identification method and device, and a mobile terminal. The method comprises: performing content identification on a user interface when an identification touch for the user interface is received; displaying an adjustable screenshot box on the user interface if the content identification of the user interface is unsuccessful; and identifying content in the screenshot box. In the method, the content identification can be directly performed on the user interface, so that operation is simple and convenient.

Description

Content recognition method, device and mobile terminal

This application claims priority from Chinese Patent Application No. 201810588338.1, filed on June 08, 2018, the entire contents of which are incorporated herein by reference.

Technical field

The present application relates to the technical field of mobile terminals, and more particularly, to a content recognition method, device, and mobile terminal.

Background technique

The display screen of the mobile terminal can display various contents. If a user wants to obtain detailed information of some of the displayed contents, the corresponding content needs to be copied to a browser search box, and the operation process is tedious.

Summary of the Invention

In view of the above problems, this application proposes a content recognition method, device, and mobile terminal, which are used to identify the content of the user interface, simplify the recognition process, and improve the user experience.

According to a first aspect, an embodiment of the present application provides a content recognition method. The method includes: when a recognition touch is received on a user interface, performing content recognition on the user interface; Failure, displaying an adjustable screenshot frame in the user interface; identifying the content within the screenshot frame.

In a second aspect, an embodiment of the present application provides a content recognition device. The device includes: a first recognition module, configured to perform content recognition on the user interface when receiving a recognition touch on the user interface; A module configured to display an adjustable screenshot frame in the user interface if the content identification of the user interface fails; a second identification module is configured to identify the content in the screenshot frame.

According to a third aspect, an embodiment of the present application provides a mobile terminal including a display screen, a memory, and a processor. The display screen and the memory are coupled to the processor. The memory stores instructions. The method described above by the processor when executed by the processor.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a processor-executable program code, where the program code causes the processor to perform the foregoing method.

The content recognition method, device and mobile terminal provided in this application display an adjustable screenshot frame and identify the content of the screenshot frame in the case that the content recognition of the user interface fails in response to the recognition touch. User interface for content identification, simple and convenient operation.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the technical solutions in the embodiments of the present application more clearly, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are just some embodiments of the application. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative labor.

FIG. 1 shows a flowchart of a content identification method according to an embodiment of the present application;

FIG. 2 shows a first display schematic diagram provided by an embodiment of the present application; FIG.

FIG. 3 shows a second display schematic diagram provided by an embodiment of the present application;

FIG. 4 shows a third display schematic diagram provided by an embodiment of the present application;

FIG. 5 is a flowchart of a content identification method according to another embodiment of the present application; FIG.

FIG. 6 shows a fourth display schematic diagram provided by an embodiment of the present application;

FIG. 7 shows a fifth display schematic diagram provided by an embodiment of the present application;

FIG. 8 shows a sixth display schematic diagram provided by an embodiment of the present application;

FIG. 9 shows a seventh display schematic diagram provided by an embodiment of the present application;

FIG. 10 shows a functional module diagram of a content recognition device according to an embodiment of the present application; FIG.

FIG. 11 shows a structural block diagram of a mobile terminal according to an embodiment of the present application;

FIG. 12 is a schematic structural diagram of a mobile terminal according to an embodiment of the present application; FIG.

FIG. 13 shows a block diagram of a mobile terminal for performing a content recognition method according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

At present, when users use the mobile terminal to chat, read text, view pictures, or watch videos on the Internet, they often become interested in some of them and search for more detailed information. At this time, the user first needs to copy the content of interest or remember the content of interest, then open the browser, and paste the copied content into the browser's search box or enter the content of the memory into the browser's search box. Searching for detailed information results in tedious, time-consuming and error-prone operations.

Further, in order to solve the problem of tedious operation process of searching, the displayed content may be selected by using pressure sensing and other technologies and the selected content may be identified to obtain a recognition result, so as to improve the speed of obtaining information. However, the inventors have discovered through a large number of studies that, in response to the user's touch on the user interface for content recognition, the recognition may fail and the recognition result desired by the user may not be obtained.

In view of the above technical problems, an embodiment of the present application proposes a content recognition method, device, and mobile terminal. In the case of recognition failure, an adjustable screenshot frame is displayed on the user interface, so that the content can be selected by re-selecting the content in the user interface. Recognize and get recognition results that fit the needs of the user.

The content identification method, device, and mobile terminal provided in the embodiments of the present application will be described below with reference to the drawings and specific embodiments.

Referring to FIG. 1, an embodiment of the present application provides a content identification method. The content identification method is used to identify all or part of content in a user interface displayed on a display screen. In a specific embodiment, the content recognition method is applied to a content recognition device as shown in FIG. 10 and a mobile terminal 400 (FIG. 11 and FIG. 12) corresponding to the content recognition device 300. The above content identification method may specifically include the following steps:

Step S110: When a recognition touch on the user interface is received, content recognition is performed on the user interface.

When a user wants to identify certain content of the user interface to obtain more detailed information of the content, the user can perform recognition touch on the user interface. The specific touch operation corresponding to the recognition touch is not limited in the embodiments of the present application, such as single-finger long press, two-finger long press, multi-finger long press, knuckle long press, single-finger tap, two-finger Tap, multi-finger tap, knuckle tap, single-finger large-area compression, two-finger large-area compression, multi-finger large-area compression, single-finger, two-finger, or multi-finger sliding on a preset trajectory, etc. If the touch operation is to slide according to a preset trajectory, the sliding trajectory may be a closed graphic to identify the content in the closed graphic.

When the mobile terminal receives the recognition touch, it recognizes the content corresponding to the recognition touch in the user interface. As a specific implementation manner, the identified display content may be all content in the user interface. In this embodiment, all content except a fixed field in the mobile terminal can be identified. If an application is displayed on the display of the mobile terminal, all content currently displayed on the display by the application is identified.

As a specific implementation manner, the identified display content may be content corresponding to the touch position of the touch operation in the user interface. The specific content corresponding to the touch position may be a text paragraph where the touch position is located, a picture where the touch position is located, and a control where the touch position is located. For example, as shown in FIG. 2, the displayed interface is a touched user interface, the circle A indicates the touch position, and the text segment where the circle A is located is used as the display content to be identified.

As a specific implementation manner, the recognized display content is a text displayed in a text control corresponding to a touch position. Specifically, the method may include: determining a text control corresponding to the touch position of the recognition touch; and acquiring text in the text control for recognition. The text control corresponding to the touch position may be a text control closest to the touch position.

That is, if the touch position is touched on the text control, the touched text control is used as the text control to be recognized. For example, in the chat interface shown in FIG. 2, a circle A indicates a touch position, and a text control B corresponding to the chat information is a text control touched by the touch position, and the chat information in the text control B is identified.

If the touch position is at a position other than the text control, the text in the text control closest to the touch position can be recognized. For example, in the chat interface shown in FIG. 3, circle A indicates a touch position, and no touch is on the text control. The text control B corresponding to the chat information is the text control closest to the touch position. Identification of chat messages.

The recognition of the displayed content may use some existing recognition methods, such as word segmentation and semantic recognition, which are not limited here.

In addition, the display content can also be identified through background screenshots. After taking screenshots of the content that needs to be identified, it can be identified through image analysis, such as OCR (Optical Character Recognition). Specifically, determining a touch position of the recognition touch in a user interface; in the user interface, taking a screenshot within a preset range of the touch position; and performing text recognition on an image obtained by the screenshot. The size of the preset range is not limited in the embodiments of the present application, and may be a rectangular range with a preset length and width, a circular range with a preset radius, or other preset shapes and a range with a preset size.

Step S120: if the content identification of the user interface fails, an adjustable screenshot frame is displayed on the user interface.

The identification of the content of the user interface may fail for various reasons. There may be multiple reasons for recognition failure, for example, network problems such as unstable network connection, mobile terminal not connected to the network, etc., lead to failure to recognize or timeout; The content to be identified is unhealthy information, etc., and is not exhaustive here.

If the recognition fails and no recognition result is obtained, an adjustable screenshot frame can be displayed in the user interface. As shown in FIG. 4, the screenshot frame K may be a rectangular frame as shown in the figure, or may be a closed shape of other shapes, such as any shape such as a circle, a prism, a triangle, and a polygon. The user interface is a user interface displayed on a display screen when a touch is recognized.

Step S130: identify the content in the screenshot frame.

As shown in FIG. 4, the content of the user interface is corresponding to the screenshot box K. The content in the screenshot box indicates the content that needs to be identified. Therefore, the content in the screenshot box can be identified.

In the embodiment of the present application, when an identification touch on the user interface is received, but the content identification of the user interface fails, but the recognition fails, the adjustable screenshot frame is displayed on the user interface, and the user interface is framed by the screenshot frame. Select, and then identify the contents of the box selection. Therefore, in the case of recognition failure, the frame selection button can be used to reframe and identify the recognition content in the user interface again.

In the embodiment of the present application, the adjustable screenshot frame can be adjusted by the user according to requirements, so that the content in the screenshot frame is the content that the user wants to identify. Specifically, referring to FIG. 5, the method provided in the embodiment of the present application includes:

Step S210: When a recognition touch on the user interface is received, content recognition is performed on the user interface.

The user can initiate a recognition touch operation to identify the displayed content in the user interface. Wherein, as described above, the user interface may be a chat interface, a web interface, a video interface, a user interface of various applications, and the like, which are not limited in the embodiments of the present application. When a user touch is received, content recognition is performed on the touched user interface.

Step S220: If the content identification of the user interface fails, an adjustable screenshot frame is displayed on the user interface.

If the recognition fails, the recognition result of the content in the touched user interface is not obtained, a screenshot frame is displayed on the user interface, and the user interface is selected and identified.

As a specific implementation manner, when the adjustable screenshot frame is displayed on the user interface, as shown in FIG. 4, the interface displayed on the display screen is the user interface touched when the touch is recognized, and is displayed on the user interface. Screenshot box K.

As a specific implementation, as shown in FIG. 6, when an adjustable screenshot frame is displayed on the user interface, the touch-sensitive user interface may be reduced and displayed, and the screenshot frame K is displayed in the reduced user interface. Outside the reduced user interface, auxiliary content can be displayed. The auxiliary content can be one or more types of recognition options such as identifying QR codes, identifying products, and identifying text. The auxiliary content can be a prompt for adjusting the screenshot box. Information, such as "select an area to be identified" in FIG. 6, and the auxiliary content may also be a control button for exiting the screenshot identification.

Optionally, when the screenshot box is displayed, only the screenshot box can be operated in the user interface, and other locations cannot be operated.

In the embodiment of the present application, when the screenshot frame starts to be displayed when the recognition fails, the position where the screenshot frame is first displayed is not limited.

As an implementation manner, the screenshot frame may be displayed at a preset position in a preset size. The preset size can be a fixed size that is set in advance, or a preset size that is proportional to the user interface. The preset position may be a certain fixed position of the display screen, or may be any certain position of the touched user interface.

As an implementation manner, a display position of the screenshot frame may be determined according to a touch position that recognizes a touch. Specifically, it may be a screenshot frame that includes a touch position frame. Optionally, the screenshot frame may be a preset size, or a minimum screenshot frame including a touch area frame corresponding to a touch position.

In this embodiment, if the touch position is on the control, the screenshot frame may include the control frame touched by the touch position. Specifically, a control corresponding to the touch position for identifying the touch may be determined, and the correspondence may indicate that in a user interface, the position where the control is located overlaps the touch position. Then, an adjustable screenshot frame including the control frame is displayed on the user interface.

In this embodiment, optionally, if the touch position is on the control, the screenshot frame may be a screenshot frame that is larger than the control, including the touched control frame. It can also be the smallest screenshot box that can select the control box where the touch position is located, that is, the screenshot box only selects the control box that it touches. The screenshot box can be a preset shape, such as a rectangle, A rounded rectangle or a shape that conforms to the shape and outline of the control where the touch location is located.

Optionally, before the screenshot box is displayed, a prompt message may be displayed to prompt the user if the recognition fails, whether to enter the box selection recognition. If the user selects Yes, an adjustable screenshot box is displayed in the user interface; if the user selects No, you can exit During the recognition process, the recognition of the user interface is ended, and no recognition result is obtained.

Step S230: identify the content in the screenshot frame.

In the embodiment of the present application, the content selected by the screenshot frame can be identified. The identification may be directly obtaining the content in the screenshot box, such as including a text control in the screenshot box, obtaining the text in the text control, including a picture control in the screenshot box, and directly obtaining a picture in the picture control. In addition, the recognition may also be that the content selected by the screenshot frame is taken as an image, for example, the edge of the screenshot frame is used as the interception edge, and all the content in the screenshot frame is taken to obtain the intercepted image, and then the Content is identified. The recognition process can first obtain the content in the picture through image processing, such as OCR (Optical Character Recognition, Optical Character Recognition) processing, etc. There is no restriction here, and the text, pictures, and two-dimensional code in the image can be obtained through the existing And so on.

As a specific implementation manner, the user interface displays a screenshot frame to identify all or part of the content in the screenshot frame.

As a specific implementation manner, when the screenshot frame is displayed, one or more optional types of identification types are provided at the same time, and the identification type indicates the type to which the content to be identified belongs, such as a QR code, a product, a text, a picture, and the like . The mobile terminal may receive a target recognition type selected by the user from one or more recognition types; and identify content corresponding to the target recognition type in the screenshot frame. That is, a recognition type selected by the user from one or more recognition types is received, and the recognition type selected by the user is used as the target recognition type to identify the content in the screenshot frame that belongs to the target recognition type. Figure 6 shows three optional recognition types: QR code, product, and text.

In this embodiment, the recognition of different recognition types may be different in recognition content. For example, text recognition only parses the text content in the screenshot box to obtain the recognition result, and picture recognition only parses the picture in the screenshot box to obtain the recognition result. result. It can also be a different recognition method, such as recognition of text, pictures, and two-dimensional codes, which is implemented by the corresponding recognition processing server. If the selected target recognition type is text, the content in the screenshot box is sent to a dedicated text The recognized server, or the text in the screenshot box is sent to a server dedicated to text recognition, which recognizes the text in the screenshot box; if the selected target recognition type is picture, the content in the screenshot box is sent To a server dedicated to image recognition, or to send a picture in a screenshot box to a server dedicated to picture recognition, and the server recognizes the picture in the screenshot box; the identification of the product can be linked to a third party Shopping platforms, such as Taobao, pass the content in the screenshot box to a third-party shopping platform for identification, in order to obtain product information and purchase links from the third-party shopping platform. The recognition results may also be displayed differently. For example, the recognition of text, products, pictures, etc. can be displayed directly through cards in the form of word segmentation, introduction, and links. The recognition of products can be displayed through third-party shopping platforms.

Specifically, the recognition process may be analyzing the content in the screenshot frame, obtaining the content corresponding to the target recognition type from it, and identifying the content of the target recognition type. For example, if the target recognition type is a two-dimensional code, analyze the content in the screenshot box to obtain the two-dimensional code therein, and then identify the two-dimensional code to obtain the information contained in the two-dimensional code. For another example, if the target recognition type is text, perform text segmentation, parsing, and semantic search operations on the text in the screenshot box, and feedback the recognition result of the text.

Among them, optionally, when the target recognition type is text, the text content can be obtained by analyzing the screenshot box, and then the text content is filtered. After filtering out the garbled characters, valid text is obtained, and then the valid text is obtained. Perform analytical identification. In this embodiment, the garbled text may be text other than a preset type of text. For example, if the preset type of text is Chinese characters, English characters, and selected common punctuation, other characters, punctuation other than common punctuation, etc. Judged as garbled.

In this embodiment, optionally, if the user has not selected the target recognition type, one of the recognition types may be used as the default recognition type to identify the content of the default recognition type in the screenshot box. Optionally, if the user does not select the target recognition type, the recognition may not be performed. After the user selects the recognition type, the content corresponding to the selected recognition type is recognized. Optionally, if the user has not selected the target recognition type, all types can be recognized.

In the embodiment of the present application, the screenshot frame is an adjustable screenshot frame, and the adjustment includes adjustment of position and adjustment of size. The mobile terminal may receive adjustments to the size or position of the screenshot frame; and identify the content within the adjusted screenshot frame.

For example, Figure 4 and Figure 6 show the selection of the screenshot frame in the user interface after entering the frame selection interface. The user can adjust the size of the screenshot frame by pulling the corner of the screenshot frame in different directions. The screenshot box K after being reduced in FIG. 4. The user can also change the position of the screenshot frame by dragging the screenshot frame by pressing the border of the screenshot frame, the inner area, etc., as shown in FIG. 8, which is adjusted relative to the position of FIG. 7.

In the embodiment of the present application, the shape of the screenshot frame can also be changed, and a user's request for changing the shape of the screenshot frame is received. According to the request, the shape of the screenshot frame is changed to a circle, a triangle, or any other polygon. The specific change of the shape can be determined by the user, that is, the shape of the screenshot frame can be changed to the shape specified by the change request. For example, each time the user clicks the button of a change request to change the shape, the shape of the screenshot box is changed in the preset shape order; or when the user's change request is received, the displayable shape options are displayed. The selected shape is used as the shape of the screenshot box to meet the needs of different users. For example, a shape selection button may be provided corresponding to the screenshot frame K, and when the button is pressed, a selectable shape is displayed. If the user selects a triangle, the shape of the current screenshot frame is changed to a triangle.

As a specific implementation manner, in the embodiment of the present application, when the screenshot frame is displayed in response to the recognition touch recognition failure, if the adjustment of the screenshot frame has not been received, the display of the screenshot frame is maintained, and the identification is not performed for the time being. , Or after waiting for the preset time, if the adjustment of the screenshot box has not been received, the content in the screenshot box is identified; after receiving the adjustment of the screenshot box, the content in the adjusted screenshot box is identified Recognize and get the recognition result.

Optionally, in this embodiment, after the adjustment of the screenshot frame is received, that is, each time the adjustment ends, the content in the screenshot frame can be identified.

Alternatively, optionally, the user may make multiple adjustments to the screenshot frame, that is, one adjustment has not reached the adjustment result desired by the user, and multiple adjustments are required. After each adjustment, wait for a preset time. If the adjustment of the screenshot frame is not received within the preset time, the content in the screenshot frame starts to be identified. If the adjustment of the screenshot frame is received within a preset time, the shape or position of the screenshot frame is changed in response to the adjustment operation. It can be understood that the preset time should not be too long, for example, it can be 1 second. Specifically, a timer can be set. When the screenshot box starts to be displayed, the timer starts timing. If the timer does not receive the adjustment of the screenshot frame when the timer reaches the preset time length, the timer is set to zero to identify the content in the screenshot frame. If an adjustment operation on the screenshot frame is received before the timer reaches the preset time length, the timer is set to zero, and the screenshot frame is adjusted according to the adjustment operation. After the operation of adjusting the screenshot frame ends, the timer starts counting. When the timer does not receive the adjustment of the screenshot frame when it reaches the preset time length again, the timer is set to zero to identify the contents of the screenshot frame; if the timer receives the time before the preset time length is received again When the screenshot frame is adjusted, the timer is set to zero, and the screenshot frame is adjusted according to the adjustment operation. After the adjustment operation of the screenshot frame is finished, the timer is counted, and so on. The timer of the controller determines whether to start recognition.

The end of the adjustment operation of the screenshot frame may be the end of the touch on the adjustment. For example, when a finger touches the border of the screenshot frame on the display screen, the adjustment of the screenshot frame starts, and the size of the screenshot frame is adjusted according to the dragging of the border line, during which the finger keeps in contact with the display screen; when it is determined that the finger is away Display, the adjustment is over.

As a specific implementation manner, when the screenshot frame is displayed in response to the failure of identifying touch recognition, the content in the screenshot frame is identified to obtain a recognition result. If the adjustment of the screenshot frame is received, the content of the adjusted screenshot frame is identified, and the recognition result is updated according to the identification of the adjusted screenshot frame. That is, as shown in FIG. 5, in this embodiment, after step S230, it may further include step S240: receiving adjustment to the size or position of the screenshot frame. Step S250: Perform screenshot identification on the adjusted screenshot frame. Optionally, in this embodiment, in the process of readjusting the size or position of the screenshot frame to update the recognition result, whether to start recognition can also be determined by the timer. For example, if an adjustment operation on the screenshot frame is received, after one adjustment operation ends, the timer starts counting from zero. If the timer again receives the adjustment operation of the screenshot frame before the preset time length expires, the timer is set to zero, the screenshot frame is adjusted according to the adjustment operation, and the adjustment operation of the screenshot frame ends. After that, the timer starts to count; if the timer does not receive adjustments to the screenshot frame when the timer reaches a preset length of time, the timer is set to zero to identify the content in the screenshot frame.

As a specific implementation manner, whether the recognition is started or not can be determined by a user. Specifically, after receiving a command related to recognition start, recognition may be started. For example, after the user adjusts the screenshot frame, it is determined that the content in the screenshot frame includes the content to be identified, and a command to start the identification is issued, and the mobile terminal starts the identification. Receiving the relevant command for the start of recognition may be: receiving a selection of a target type; providing a touch button dedicated to recognize the start, receiving a touch to the touch button; or receiving a gesture corresponding to the start of recognition Operations, such as clicking or double-clicking in the screenshot box.

In the embodiment of the present application, the recognition in the screenshot box may include word segmentation and corresponding search for the corresponding content, and the recognition result may include the word segmentation result, the introduction, link, location map of the film and television, books, characters, etc., and the purchase channel of the product. One or more of schedule information, courier information, and the like are not limited in the embodiments of the present application, and may be any interpretation information of the displayed content. FIG. 9 shows a specific manner of displaying a recognition result. The displayed recognition result may include segmentation of corresponding content. The user may select a word from the segmentation result and then copy, select all, translate, or search.

Optionally, in the embodiment of the present application, the recognition result may be displayed by a card. As shown in FIG. 9, the card C displays the recognition result. The card is a carrier for displaying information, and may be a control or a combination of multiple controls. The information displayed by the card in the embodiment of the present application may be information corresponding to the recognition result. In the same card, different recognition results of the same display content can be displayed, or different recognition results of the same display content can be displayed on different cards.

Optionally, in the embodiment of the present application, if the content in the screenshot frame is unsuccessfully identified, a prompt message that the recognition fails may be displayed.

In summary, in the content recognition method provided in the embodiment of the present application, when a recognition touch is received, content recognition is performed on a user interface. If the identification fails, a screenshot frame is provided for selecting the user interface. When receiving adjustments to the size or position of the screenshot box, the content of the adjusted screenshot box is identified, so that users can adjust the size and position of the screenshot box according to their own needs, and select the content box that they want to identify in the screenshot Recognize in the frame to get the desired recognition result.

In the embodiment of the present application, the above-mentioned various implementation manners can be arbitrarily combined under logical circumstances, and the embodiments of the present application will not go into details of the various combining schemes.

An embodiment of the present application further provides a content recognition device 300. Referring to FIG. 10, the device 300 includes: a first recognition module 310 configured to perform content processing on the user interface when receiving a recognition touch on the user interface. Identify. A frame selection module 320 is configured to display an adjustable screenshot frame on the user interface if the content identification of the user interface fails. The second identification module 330 is configured to identify content in the screenshot frame.

Optionally, the first recognition module 310 may include a control determining unit for determining a text control corresponding to a touch position for recognizing a touch, and a recognition unit for acquiring text in the text control for recognition.

Optionally, the first recognition module may include a position determination unit for determining a touch position for recognizing a touch, and an image acquisition unit for presetting the touch position in the user interface. Take a screenshot within the scope; a recognition unit, configured to perform text recognition on the picture obtained by the screenshot.

Optionally, the device may further include: a type determination module configured to receive a target recognition type selected by the user from one or more recognition types; a second recognition module configured to correspond to the content corresponding to the target recognition type in the screenshot frame For identification.

Optionally, the device may further include: an adjustment module, configured to receive adjustment of the size or position of the screenshot frame. The second recognition module is further configured to perform screenshot recognition on the adjusted screenshot frame.

Optionally, the frame selection module 320 may be further configured to display the screenshot frame at a preset position in a preset size. Alternatively, it is used to determine a control corresponding to the touch position for identifying touch; and an adjustable screenshot frame including the control frame is displayed on the user interface.

In summary, in the embodiment of the present application, when the user is chatting or viewing text in the browser, after the text in the area selected by the user fails to be recognized, a manual screenshot box is displayed so that the user can manually select the area that needs to be captured. Then, the captured picture is identified. When the screenshot picture is identified, the user can select whether it is a two-dimensional code recognition, a text recognition, or an article recognition. Among them, preliminary analysis can be performed on the parsed results, and garbled characters and characters are filtered. If there is no valid text after filtering, a process that does not recognize any content is taken, and if there is valid text after filtering, it is passed to text recognition.

Please refer to FIG. 11 again. Based on the foregoing content identification method and device, an embodiment of the present application further provides a mobile terminal 400. As shown in FIG. 11, the mobile terminal 400 includes a display screen 120, a memory 104, and a processor 102. The display screen 120 and the memory 104 are coupled to the processor 102. The display screen 120 is used to display content. , Parsing recognition results, etc., the memory 104 stores instructions, and when the instructions are executed by the processor 102, the processor 102 executes the method provided in the embodiment of the present application.

Specifically, as shown in FIG. 12, the mobile terminal 400 may include an electronic body portion 10, and the electronic body portion 10 includes a casing 12 and a display screen 120 disposed on the casing 12. The casing 12 can be made of metal, such as steel and aluminum alloy. In this embodiment, the display screen 120 generally includes a display panel 111, and may also include a circuit for responding to a touch operation on the display panel 111, and the like. The display panel 111 may be a liquid crystal display (Liquid Crystal Display, LCD). In some embodiments, the display panel 111 is a touch screen 109 at the same time.

Please refer to FIG. 13 at the same time. In an actual application scenario, the mobile terminal 400 can be used as a smart phone terminal. In this case, the electronic body portion 10 usually further includes one or more (only shown in the figure) (A) The processor 102, the memory 104, an RF (Radio Frequency) module 106, an audio circuit 110, a sensor 114, an input module 118, and a power module 122. A person of ordinary skill in the art can understand that the structure shown in FIG. 13 is only for illustration, and it does not limit the structure of the electronic body portion 10. For example, the electronic body portion 10 may further include more or fewer components than those shown in FIG. 13, or have a different correspondence from that shown in FIG. 13.

Those of ordinary skill in the art can understand that, with respect to the processor 102, all other components are peripherals, and the processor 102 and these peripherals are coupled through multiple peripheral interfaces 124. The peripheral interface 124 may be implemented based on the following standards: Universal Asynchronous Receiver / Transmitter (UART), General Input / Output (GPIO), Serial Peripheral Interface , SPI), Inter-Integrated Circuit (I2C), but not limited to the above standards. In some examples, the peripheral interface 124 may only include a bus; in other examples, the peripheral interface 124 may further include other elements, such as one or more controllers, for example, for connecting the display panel. A display controller of 111 or a memory controller for connecting a memory. In addition, these controllers can also be separated from the peripheral interface 124 and integrated into the processor 102 or a corresponding peripheral.

The memory 104 may be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104. The memory 104 may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memories remotely disposed with respect to the processor 102, and these remote memories may be connected to the electronic body portion 10 or the display screen 120 through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The RF module 106 is used to receive and send electromagnetic waves, to realize mutual conversion between electromagnetic waves and electrical signals, and to communicate with a communication network or other equipment. The RF module 106 may include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and the like . The RF module 106 can communicate with various networks such as the Internet, an intranet, and a wireless network, or communicate with other devices through a wireless network. The wireless network may include a cellular telephone network, a wireless local area network, or a metropolitan area network. The above wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data Communication Technology (GSM) Environment, EDGE, Broadband Code Division multiple access technology (wideband code division multiple access, W-CDMA), code division multiple access technology (Code division access, CDMA), time division multiple access technology (time division multiple access, TDMA), wireless fidelity technology (Wireless, Fidelity , WiFi) (such as the American Institute of Electrical and Electronics Engineers standards IEEE 802.10A, IEEE 802.11b, IEEE802.11g, and / or IEEE 802.11n), Voice over Internet (Internet Protocol, VoIP), Global Microwave Interoperability (Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for mail, instant messaging, and short messaging, and any other suitable communication protocol, even those that have not yet been developed.

The audio circuit 110, the speaker 101, the microphone 103, and the microphone 105 collectively provide an audio interface between the user and the electronic body portion 10 or the display screen 120.

The sensor 114 is disposed in the electronic body portion 10 or the display screen 120. Examples of the sensor 114 include, but are not limited to, an acceleration sensor 114F, a gyroscope 114G, a magnetometer 114H, and other sensors.

In this embodiment, the input module 118 may include the touch screen 109 provided on the display screen 120, and the touch screen 109 may collect a touch operation performed by the user on or near the touch screen (such as a user using a finger, a stylus, etc.) Any suitable object or accessory is operated on or near the touch screen 109), so that the user ’s touch gesture can be obtained, and the corresponding connection device is driven according to a preset program, so the user can The touch operation of the display selects the target area. Optionally, the touch screen 109 may include a touch detection device and a touch controller. The touch detection device detects a user's touch position, and detects a signal caused by a touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, and The touch information is converted into touch point coordinates, and then sent to the processor 102, and can receive and execute commands sent by the processor 102. In addition, various types such as resistive, capacitive, infrared, and surface acoustic wave can be used to implement the touch detection function of the touch screen 109. In addition to the touch screen 109, in other modified embodiments, the input module 118 may further include other input devices, such as keys 107. The keys 107 may include, for example, character keys for inputting characters, and control keys for triggering control functions. Examples of the control buttons include a "return to the home screen" button, an on / off button, and the like.

The display screen 120 is used to display information input by the user, information provided to the user, and various graphical user interfaces of the electronic body portion 10, and these graphical user interfaces may include graphics, text, icons, numbers, videos, and other Composed of any combination. In one example, the touch screen 109 may be disposed on the display panel 111 so as to form a whole with the display panel 111.

The power module 122 is configured to provide power to the processor 102 and other components. Specifically, the power module 122 may include a power management system, one or more power sources (such as a battery or AC power), a charging circuit, a power failure detection circuit, an inverter, a power status indicator, and any other electronic components Components related to the generation, management and distribution of power in the display 10 or the display 120.

The mobile terminal 400 further includes a locator 119, which is configured to determine an actual location where the mobile terminal 400 is located. In this embodiment, the locator 119 uses a positioning service to implement positioning of the mobile terminal 400. The positioning service should be understood as obtaining position information (such as latitude and longitude coordinates) of the mobile terminal 400 through a specific positioning technology. ), A technology or service for marking the location of an object on an electronic map.

It should be understood that the above-mentioned mobile terminal 400 is not limited to a smart phone terminal, and it should refer to a computer device that can be used in mobile. Specifically, the mobile terminal 400 refers to a mobile computer device equipped with a smart operating system. The mobile terminal 400 includes, but is not limited to, a smart phone, a smart watch, a tablet computer, and the like.

It should be noted that each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same and similar parts between the embodiments, refer to each other. can. As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant part, refer to the description of the method embodiment. Any of the processing methods described in the method embodiments can be implemented by corresponding processing modules in the device embodiments, and will not be described in detail in the device embodiments.

In the description of this specification, the description with reference to the terms “one embodiment”, “some embodiments”, “examples”, “specific examples”, or “some examples” and the like means specific features described in conjunction with the embodiments or examples , Structure, materials, or features are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, without any contradiction, those skilled in the art may combine and combine different embodiments or examples and features of the different embodiments or examples described in this specification.

In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "plurality" is at least two, for example, two, three, etc., unless it is specifically and specifically defined otherwise.

Any process or method description in a flowchart or otherwise described herein can be understood as representing a module, fragment, or portion of code that includes one or more executable instructions for implementing a particular logical function or step of a process And, the scope of the preferred embodiments of the present application includes additional implementations, in which the functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application pertain.

The logic and / or steps represented in the flowchart or otherwise described herein, for example, a sequenced list of executable instructions that can be considered to implement a logical function, can be embodied in any computer-readable medium, For the instruction execution system, device, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from the instruction execution system, device, or device), or combine these instruction execution systems, devices, or devices Or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connections (mobile terminals) with one or more wirings, portable computer disk enclosures (magnetic devices), random access memory (RAM), Read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, because, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or other suitable Processing to obtain the program electronically and then store it in computer memory.

It should be understood that each part of the application may be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it may be implemented using any one or a combination of the following techniques known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

A person of ordinary skill in the art can understand that all or part of the steps carried by the methods in the foregoing embodiments may be implemented by a program instructing related hardware. The program may be stored in a computer-readable storage medium. The program is When executed, one or a combination of the steps of the method embodiment is included. In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist separately physically, or two or more units may be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application. Those skilled in the art can interpret the above within the scope of the present application. Embodiments are subject to change, modification, substitution, and modification.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still Modifications to the technical solutions described in the foregoing embodiments, or equivalent replacements of some of the technical features thereof; and these modifications or replacements do not drive the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

A content identification method, characterized in that the method includes:

When a recognition touch to the user interface is received, performing content recognition on the user interface;

If the content identification of the user interface fails, an adjustable screenshot frame is displayed on the user interface;

Identify the content in the screenshot box.
The method according to claim 1, wherein the sliding track of the touch operation corresponding to the recognition touch is a closed graphic.
The method according to claim 1 or 2, wherein the performing content identification on the user interface comprises:

Determining a text control corresponding to the touch position of the recognition touch;

Acquire text in the text control for recognition.
The method according to claim 3, wherein the determining a text control corresponding to a touch position for recognizing a touch comprises:

If the touch position is on a text control, using the text control where the touch position is as a text control corresponding to the touch position;

If the touch position is not on the text control, the text control closest to the touch position is used as the text control corresponding to the touch position.
The method according to any one of claims 1-4, wherein the performing content identification on the user interface comprises:

Determining a touch position of the recognition touch;

Taking a screenshot in a preset range of the touch position in the user interface;

Text recognition is performed on the picture obtained by the screenshot.
The method according to any one of claims 1-5, wherein the displaying an adjustable screenshot frame on the user interface comprises:

Reducing the display of the user interface, and displaying the screenshot frame in the reduced user interface;

Outside the reduced user interface, auxiliary content is displayed, the auxiliary content including one or more recognition types.
The method according to any one of claims 1-5, further comprising:

Receiving a target recognition type selected by the user from one or more recognition types;

The content corresponding to the target recognition type in the screenshot box is identified.
The method according to claim 7, wherein the identification type comprises:

QR code, product or text.
The method according to claim 7 or 8, wherein identifying the content corresponding to the target recognition type in the screenshot frame comprises:

If the target recognition type is text, sending the content in the screenshot box to a server dedicated to text recognition, and the server recognizes the text in the screenshot box;

If the target recognition type is a picture, sending the content in the screenshot box to a server dedicated to picture recognition, and the server recognizes the picture in the screenshot box;

If the target identification type is a product, the content in the screenshot box is passed to a third-party shopping platform for identification, so as to obtain product information and a purchase link from the third-party shopping platform.
The method according to any one of claims 1-9, wherein a timer is set in advance, and the identifying the content in the screenshot frame comprises:

When the screenshot box starts to display, the timer starts counting;

If an adjustment operation on the screenshot frame is received before the timer reaches the preset time length, the timer is set to zero, and the screenshot frame is adjusted according to the adjustment operation.

After the operation of adjusting the screenshot frame ends, the timer starts counting;

If the timer does not receive the adjustment of the screenshot frame when the timer reaches the preset time length, the timer is set to zero to identify the content in the screenshot frame.
The method according to any one of claims 1 to 10, wherein after identifying the content in the screenshot frame, further comprising:

Receiving adjustments to the size or position of the screenshot frame;

Perform screenshot recognition on the adjusted screenshot frame.
The method according to any one of claims 1-11, wherein the displaying an adjustable screenshot frame on the user interface comprises:

Displaying the screenshot frame at a preset position in a preset size and a preset shape.
The method according to any one of claims 1-11, wherein the displaying an adjustable screenshot frame on the user interface comprises:

Determining a control corresponding to the touch position of the recognition touch;

An adjustable screenshot frame selected from the control frame is displayed on the user interface.
The method according to claim 13, wherein the displaying on the user interface an adjustable screenshot frame including the control frame selection comprises:

Shows the smallest screenshot box with the control box where the touch position is located.
The method according to claim 14, wherein a smallest screenshot frame including a touch area frame corresponding to the touch position is displayed, and a shape of the screenshot frame is consistent with a shape contour of a control where the touch position is located.
The method according to any one of claims 12 to 14, wherein after displaying an adjustable screenshot frame in the user interface, the method further comprises

Receiving a change request for changing the shape of the screenshot frame;

Changing the shape of the screenshot frame to a shape specified by the change request.
The method according to any one of claims 1-16, wherein the identifying the content in the screenshot frame comprises:

Analyze the screenshot box to obtain the text content;

After filtering garbled characters in the text content, valid text is obtained, the garbled text is text other than a preset type of text, and the preset type of text is Chinese characters, English characters, and selected common punctuation;

Analyze and identify the valid text.
A content recognition device, characterized in that the device includes:

A first recognition module, configured to perform content recognition on the user interface when receiving a recognition touch on the user interface;

A frame selection module, configured to display an adjustable screenshot frame on the user interface if the content identification of the user interface fails;

The second identification module is configured to identify content in the screenshot frame.
A mobile terminal is characterized by comprising a display screen, a memory and a processor, the display screen and the memory are coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor When the processor executes the method according to any one of claims 1 to 17.
A computer-readable storage medium having a processor-executable program code, wherein the program code causes the processor to perform the method according to any one of claims 1 to 17.