WO2025245697A1 - 利用数字助手处理图像的方法、装置、设备和存储介质 - Google Patents

利用数字助手处理图像的方法、装置、设备和存储介质

Info

Publication number
WO2025245697A1
WO2025245697A1 PCT/CN2024/095791 CN2024095791W WO2025245697A1 WO 2025245697 A1 WO2025245697 A1 WO 2025245697A1 CN 2024095791 W CN2024095791 W CN 2024095791W WO 2025245697 A1 WO2025245697 A1 WO 2025245697A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
page
processing
digital assistant
control
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/095791
Other languages
English (en)
French (fr)
Inventor
张凡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Priority to PCT/CN2024/095791 priority Critical patent/WO2025245697A1/zh
Priority to CN202480002189.XA priority patent/CN119156648A/zh
Publication of WO2025245697A1 publication Critical patent/WO2025245697A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9535—Search customisation based on user profiles and personalisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding

Definitions

  • the exemplary embodiments disclosed herein generally relate to the field of image processing, and particularly to methods, apparatus, devices, and computer-readable storage media for processing images using digital assistants.
  • Digital assistants are provided to assist users with various task processing needs in different applications and scenarios.
  • Digital assistants typically possess intelligent dialogue and task processing capabilities. They can be invoked in response to user input. For example, during an interaction with a digital assistant, the user inputs interactive messages, and the digital assistant responds by providing reply messages.
  • a method for processing images using a digital assistant includes: receiving an image captured from a target page; sending the image to the digital assistant; determining a processing intent for processing the image based on its content; and obtaining a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent.
  • an apparatus for processing images using a digital assistant includes: a receiving module configured to receive an image captured from a target page and send the image to the digital assistant; a determining module configured to determine a processing intent for processing the image based on the image's content; and an acquiring module configured to acquire a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent.
  • an electronic device in a third aspect of this disclosure, includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect of this disclosure.
  • a computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect of this disclosure.
  • a computer program product comprising a computer program wherein the computer program, when executed by a processor, implements the method of the first aspect of this disclosure.
  • Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented
  • FIGS. 2A to 2C illustrate schematic diagrams of example pages according to some embodiments of the present disclosure
  • Figure 3 illustrates a flowchart of a process for processing images using a digital assistant according to some embodiments of the present disclosure
  • Figure 4 shows a block diagram of an apparatus for processing images using a digital assistant according to some embodiments of the present disclosure.
  • Figure 5 shows a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented.
  • performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.
  • a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.
  • a prompt message in response to a user's active request, can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format.
  • the pop-up window can also include a selection control allowing the user to choose "agree” or “disagree” to provide personal information to the electronic device.
  • model learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques, such as deep learning.
  • a neural network is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs.
  • a neural network model is an example of a model based on deep learning.
  • model may also be referred to as “machine learning model,””learningmodel,””machine learning network,” or “learning network,” and these terms are used interchangeably.
  • a neural network is a machine learning network based on deep learning.
  • a neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers.
  • Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth.
  • the layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer.
  • the input layer receives the input to the neural network, while the output layer's output serves as the final output.
  • Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.
  • Machine learning typically comprises three phases: training, testing, and application (also known as inference).
  • training phase a given model is trained using a large amount of training data, iteratively updating parameter values until the model can consistently generate inferences that meet the expected goals from the training data.
  • the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data.
  • the parameter values of the trained model are determined.
  • test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance.
  • the testing phase can sometimes be integrated into the training phase.
  • the trained model can be used to process actual model inputs based on the trained parameter values to determine the corresponding model output.
  • Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented.
  • a digital assistant 120 may be installed in a terminal device 110 (or, as may be said, a digital assistant 120 is deployed in a terminal device 110).
  • a user 140 may interact with the digital assistant 120 via the terminal device 110 and/or an attached device of the terminal device 110.
  • the digital assistant 120 may be included in or implemented as a target application, which is an application already installed on the terminal device 110.
  • the digital assistant 120 may... Configured to have intelligent dialogue capabilities and task processing capabilities within the target application, the digital assistant 120 is provided to assist users with various task processing needs in different applications and scenarios.
  • Interactive messages may include multimodal messages, such as text messages (e.g., natural language text), voice messages, image messages, video messages, etc.
  • the digital assistant 120 supports users inputting questions in natural language and performs tasks and provides replies based on its understanding of natural language input and logical reasoning capabilities.
  • the target application may present a corresponding page 150 to user 140 based on user 140's actions to output and/or receive information from user 140.
  • the implementation of at least some functions of the target application, and/or the implementation of at least some functions of the digital assistant 120 in the target application may be based on a target model.
  • the target model may be a machine learning model, a deep learning model, a learning model, a neural network, etc.
  • the model may be based on a language model (LM).
  • LM language model
  • the target model may also be based on other suitable models.
  • terminal device 110 communicates with server 130 to provide services to target applications and/or digital assistant 120.
  • Terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio/video players, digital cameras/camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.
  • terminal device 110 can also support any type of user-facing interface (such as "wearable" circuitry).
  • Server 130 can be various types of computing systems/servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.
  • an improved scheme for processing images using a digital assistant is provided.
  • an image captured from a target page is received and sent to a digital assistant.
  • a processing intent for processing the image is determined.
  • a processing result associated with the image is obtained, the processing result being determined by the digital assistant based on the processing intent.
  • the digital assistant can process the captured image based on the processing intent and present the processing result in a dialog page. This helps improve the efficiency of image processing, reduces the difficulty for users to obtain information, and improves the efficiency of users obtaining information.
  • the method for processing images using a digital assistant can be implemented as a plugin for the digital assistant, and the method is invoked in response to the activation of the plugin.
  • the digital assistant can provide a switch indicating whether the method is enabled.
  • the method for processing images using the digital assistant can be executed only when the switch is enabled.
  • FIGS 2A to 2C illustrate schematic diagrams of example pages 200A to 200C (also referred to simply as examples 200A to 200C) according to some embodiments of the present disclosure. It should be understood that the pages shown in the figures are merely examples, and various page designs may actually exist. The various graphic elements on the page may have different arrangements and different visual representations, one or more elements may be omitted or replaced, and one or more other elements may also be present. The embodiments of the present disclosure are not limited in this respect.
  • Examples 200A to 200C can be displayed on terminal device 110.
  • Examples 200A to 200C will be described with reference to environment 100 of Figure 1. It should be noted that some operations described in this document with reference to terminal device 110 may require the assistance of server 130. Furthermore, the operations performed by terminal device 110 may specifically be performed by relevant applications and/or digital assistant 120 installed on terminal device 110.
  • terminal device 110 receives an image captured from a target page.
  • the target page can be any suitable page from which content can be captured, including but not limited to document pages, web pages, video pages, media item browsing pages, message pages, etc.
  • terminal device 110 can receive an image 201 captured from target page 210
  • terminal device 110 can receive an image 202 captured from target page 210.
  • image 201 can be an image captured by calling the screenshot function of the operating system, and/or an image captured by calling the screenshot function of other applications.
  • terminal device 110 sends an image to digital assistant 120 for processing by digital assistant 120 (this process may also be referred to as invoking digital assistant 120 to process the image).
  • terminal device 110 may provide invocation controls (e.g., the various controls in invocation page 220 shown in FIG. 2B). Invocation controls may be provided in response to the acquisition of an image (i.e., the image captured from the target page as described above), or may be displayed by default, for example, floating on top of the target page.
  • Terminal device 110 may, for example, send an image to digital assistant 120 for processing by digital assistant 120 in response to receiving a user operation on the invocation control (e.g., a user trigger operation on the invocation control).
  • a user operation on the invocation control e.g., a user trigger operation on the invocation control
  • terminal device 110 may also provide a control page for invoking digital assistant 120 to process images, and in response to receiving a first interaction request in the control page, invoking digital assistant 120 to process images.
  • the control page may, for example, be a floating page displayed on top of the target page.
  • terminal device 110 may provide control page 220, which is a floating page displayed on top of target page 210.
  • Control page 220 may, for example, include multiple interactive controls, including at least one interactive control with the function of invoking digital assistant (e.g., interactive controls 222, 223, and 224 in example 200A, and interactive controls 225 and 226 in example 200B).
  • Terminal device 110 may respond to receiving a request for a target interactive control.
  • the first interaction request (e.g., a user action on the target control) of any interactive control that invokes the digital assistant invokes the digital assistant 120 to process the image.
  • the processing intent for processing an image can be determined based on its content. For example, if the image contains text content, the processing intent could be determined as automatic recognition, automatic translation, automatic search, etc. Specifically, more refined processing intents can be determined based on text recognition; for example, if the image includes names and contact information, the processing intent could be determined as adding a contact; if the image includes time, location, and other information, the processing intent could be determined as creating a schedule, and so on. At this point, the digital assistant 120 can be invoked to determine the processing result.
  • the processing intent can be determined to be: querying detailed information, purchasing, etc. Then, the digital assistant 120 can be invoked to provide the corresponding query results or purchase links, etc.
  • a dialog page may be provided, which is a conversational interaction page of the digital assistant, and the dialog page includes an image, a processing intent, and a processing result.
  • terminal device 110 may provide a dialog page in response to the invocation of digital assistant 120, the dialog page being a conversational interaction page of the digital assistant, and the dialog page including an image.
  • terminal device 110 may, for example, present dialog page 230 when digital assistant 120 is invoked to process an image.
  • Dialogue page 230 is a dialogue page between a user (e.g., user 140) and digital assistant 120.
  • Dialogue page 230 may also be, for example, a floating page displayed on top of target page 210.
  • the dialog page 230 may include an image 233, a processing intent (e.g., a conversation message 234 indicating the processing intent is to find a relevant webpage), and a processing result (e.g., a conversation message 235 indicating a response from the digital assistant).
  • a processing intent e.g., a conversation message 234 indicating the processing intent is to find a relevant webpage
  • a processing result e.g., a conversation message 235 indicating a response from the digital assistant.
  • the user may copy the image to the dialog page; alternatively and/or additionally, the image 233 may be automatically displayed on the dialog page, thereby further simplifying user operations and improving the efficiency of information retrieval.
  • multiple processing intentions can be determined from various aspects for a given image.
  • the processing intentions include multiple processing intentions
  • the dialog page includes multiple processing intentions, each corresponding to one of the multiple processing intentions.
  • a prompt is provided.
  • the user can select the desired processing intent from multiple prompts, thereby specifying the task of the digital assistant.
  • the processing result is obtained based on the target processing intent corresponding to the target prompt among the multiple processing intents.
  • processing intentions can be identified: automatic recognition, automatic translation, automatic search, etc. Multiple prompts corresponding to these processing intentions can be provided, and the image can be processed based on the user's selected prompt. If the user selects "automatic recognition,” the processing result can represent the text recognition result. If the user selects "automatic translation,” the target language can be based on the languages involved in the user's historical interactions with the digital assistant; alternatively and/or additionally, the user can be further prompted to specify the target language, and a corresponding translation result can be provided. If the user selects "automatic search,” results corresponding to the text content can be searched within a specified range. In this case, the results can involve various media formats, such as text, images, audio, video, etc.
  • obtaining the processing result associated with an image may further include: obtaining contextual data of the image from the target page, and then obtaining the processing result associated with the image based on the contextual data. For example, if a user has extracted an image from a paper, the text content associated with the image in the paper can be used as the image's contextual data. Furthermore, the contextual data can be used to analyze the specific meaning represented by the image. If the image includes multiple curves with multiple parameters, the digital assistant can use the specific definitions of each parameter in the contextual data to interpret the meaning of each curve, the relationship between multiple parameters, and so on. Similarly, if a user has extracted a photo from a news webpage, the digital assistant can use the text report in the news webpage as contextual data and interpret the identities of the people in the photo, and so on.
  • terminal device 110 may respond to receiving a move request for moving either a control page or a dialog page, and move to the location of the page specified by the move request. For example, terminal device 110 may determine that a move request for a control page and/or dialog page has been received in response to receiving a drag operation for the control page and/or dialog page. Terminal device 110 may, for example, determine the location where the drag operation ended as the location specified by the move request. A fixed location. Alternatively and/or additionally, dialog page 230 can also be displayed at a fixed location.
  • terminal device 110 may, in response to determining that dialog page 230 is to be presented, or in response to determining that dialog page 230 should be presented, no longer present (i.e., hide) control page 220.
  • terminal device 110 may present dialog page 230 on top of target page 210 while maintaining the presentation of control page 220 (i.e., control page 220 and dialog page 230 may be presented together).
  • Dialog page 230 includes a cancel control 232.
  • Terminal device 110 may cancel the presentation of dialog page 230 in response to receiving a user operation on cancel control 232.
  • Dialog page 230 presents image 233, which may, for example, be an image previously captured by terminal device 110 from target page 210.
  • Image 233 may be presented in dialog page 230 in the form of a session message from the user.
  • control page 220 includes a first interactive control for performing a first predetermined task.
  • control page 220 may include an interactive control 222 for performing a detailed description task and an interactive control 223 for performing a web page search task.
  • control page 220 may include an interactive control 225 for performing a term viewing task and an interactive control 226 for performing a search for related content task.
  • Terminal device 110 may, in response to receiving a second interaction request for a first interactive control, provide a first result in a dialog page.
  • This first result is the result of digital assistant 120 performing a first predetermined task on an image.
  • the first predetermined task may be a pre-specified task.
  • terminal device 110 may, in response to receiving a second interaction request for interactive control 222, perform a detailed description task.
  • Dialog page 230 as shown in example 200C, may provide the first result obtained by digital assistant 120 performing a detailed description task of the content in an image, such as presenting text and/or other information extracted from the image, etc.
  • terminal device 110 may receive a third interaction request for adding a second interactive control to a control page. In some embodiments, terminal device 110 may determine that it has received a third interaction request for adding a second interactive control to a control page in response to receiving a user operation for adding a control.
  • the control page may include a first add control (e.g., the add control in control page 220 shown in Figures 2A and 2B). (Item 221)
  • terminal device 110 can determine that it has received a third interaction request for adding a second interactive control to the control page. In this way, users can add personalized controls according to their own needs.
  • the dialog page may include settings controls (e.g., settings control 231 in dialog page 230 of FIG2C), and the terminal device 110 may, for example, present the settings page in response to receiving a user operation on the settings control.
  • the settings page may, for example, include a second add control.
  • the dialog page may also directly include the second add control.
  • the terminal device 110 may, in response to receiving a user operation on the second add control, determine that a third interaction request for adding a second interactive control to the control page has been received.
  • terminal device 110 may, in response to receiving a user operation on the first added control, determine that a third interaction request for adding a second interactive control to the control page has been received.
  • Terminal device 110 may also, in response to receiving a user operation on the first added control, present a settings page for specifying a second predetermined task.
  • Terminal device 110 may receive user input via this page and determine the second predetermined task based on the user input.
  • the predetermined task may include, but is not limited to, extracting text from an image, recognizing objects in an image, etc. In this way, it is easier for users to add controls to the control page, thereby performing the predetermined task in a simpler manner.
  • the third interaction request may also be text input received in the dialog page, and the terminal device 110 may determine the second predetermined task based on the text input. For example, the terminal device 110 may determine the second predetermined task as task XXXX in response to receiving text such as "Add the control corresponding to task XXXX in the control page" in the dialog page.
  • the terminal device 110 can then respond to receiving a third interaction request for adding a second interactive control to a control page, determine a second predetermined task corresponding to the second interactive control, and add a second interactive control to the control page for invoking a digital assistant to perform the second predetermined task. For example, if the terminal device 110 determines that the second predetermined task is the task of recognizing a formula in an image, then the terminal device 110 can add a second interactive control to the control page for invoking a digital assistant to perform the task of recognizing a formula in an image.
  • At least one interactive control in the control page for invoking the digital assistant may include an interactive control solely for invoking the digital assistant (e.g., interactive control 224 in FIG. 2A).
  • Terminal device 110 may present a dialog page in response to the triggering of this interactive control. Further, terminal device 110 may receive user input via the dialog page. Terminal device 110 may determine the received user input as a fourth interaction request. In response to receiving the fourth interaction request for the dialog page, terminal device 110 may provide a second result in the dialog page, the second result being the result of the digital assistant performing the task specified by the fourth interaction request on the image.
  • terminal device 110 can receive user input via input box 236 and identify the user input as a fourth interaction request.
  • Terminal device 110 can present the received user input in the form of a session message from the user (e.g., session message 234 shown in the figure).
  • Digital assistant 120 can perform the task specified by the fourth interaction request.
  • Terminal device 110 can present the result obtained by digital assistant 120 in performing the task specified by the fourth interaction request, for example, in the form of a session message from digital assistant 120 (i.e., session message 235 shown in the figure) on dialog page 230.
  • the terminal device 110 may also provide controls on the dialog page to perform subsequent tasks based on the results in the dialog page (i.e., the results of the image processing by the digital assistant). These controls may include, for example, any suitable controls such as edit controls, send controls, or favorite controls.
  • the terminal device 110 may perform the subsequent task in response to receiving a fifth interaction request for the control. For example, the terminal device 110 may add the result to the user's favorites in response to receiving a user action on the favorite control.
  • the terminal device 110 may also provide data associated with the video clip.
  • the terminal device 110 may instruct the digital assistant 120 to determine which movie the image 202 is from and which character in the movie it is, and provide information such as movie introduction, character introduction, and movie reviews associated with the movie in the dialog page.
  • terminal device 110 can display relevant historical figures, scientific explanations, or solutions to mathematical problems on a dialog page based on the content of the screenshots. This can improve the efficiency of users acquiring knowledge.
  • terminal device 110 can provide price comparison information for products on the chat page based on the content in the screenshots, and recommend personalized coupons, related product combinations or alternatives based on the user's shopping history and preferences, thereby improving shopping convenience and efficiency.
  • the terminal device 110 can provide nutritional analysis, health advice, or recipe recommendations on the chat page based on the content of the screenshots. For screenshots related to exercise, it can recommend corresponding exercise methods or provide recommendations for exercise venues near the user's location on the chat page.
  • the terminal device 110 can provide an introduction to the location in the image on the dialogue page based on the content of the screenshot, and provide detailed travel guides, real-time weather, local cultural background knowledge, travel agency recommendations, and budget planning, etc.
  • terminal device 110 can provide instant translation on the dialogue page based on the content in the screenshot. It can also provide pronunciation, grammar analysis, usage scenarios, and practice dialogues for that language, helping users improve their language learning efficiency.
  • the terminal device 110 can provide decoration suggestions on the chat page based on the content of the screenshot, such as color matching, furniture selection and space utilization solutions, and can even recommend local decoration service providers and related promotional information.
  • terminal device 110 can provide character analysis, plot discussions, behind-the-scenes information, and in-depth information on related literary works or historical events, as well as recommendations for similar film and television works, based on the content in the screenshots on the dialogue page, which can enhance the efficiency of information acquisition after watching films and TV series.
  • screenshots can be automatically sent to a digital assistant, which can then process the captured images and display them on the dialog page.
  • the processing results are displayed in the image. This helps improve the efficiency of image processing, reduces the difficulty for users to obtain information, and improves the efficiency of information retrieval for users.
  • FIG. 3 shows a flowchart of a process 300 for processing an image using a digital assistant according to some embodiments of this disclosure.
  • Process 300 can be implemented at a terminal device 110.
  • Process 300 is described below with reference to Figure 1.
  • an image captured from the target page is received and sent to the digital assistant.
  • the processing intent for processing the image is determined based on the content of the image.
  • the processing result associated with the image is obtained, which is determined by the digital assistant based on the processing intent.
  • process 300 further includes: providing a dialogue page, which is a conversational interaction page of a digital assistant, and the dialogue page includes an image, a processing intent, and a processing result.
  • the processing intent includes multiple processing intents
  • the dialog page includes multiple prompts corresponding to the multiple processing intents respectively.
  • Obtaining the processing result associated with the image further includes: in response to receiving a selection request for a target prompt among the multiple prompts, obtaining the processing result based on the target processing intent corresponding to the target prompt among the multiple processing intents.
  • obtaining the processing result associated with the image further includes: obtaining context data of the image from the target page; and obtaining the processing result associated with the image based on the context data.
  • obtaining the processing result associated with the image further includes: providing a control page for invoking a digital assistant to process the image; and invoking the digital assistant to process the image in response to receiving a first interaction request in the control page.
  • control page includes a first interactive control for performing a first predetermined task
  • providing the dialog page includes: in response to receiving a second interactive request for the first interactive control, providing a first result in the dialog page, the first result being the result of the digital assistant performing the first predetermined task on the image.
  • process 300 further includes: in response to receiving a third interaction request for adding a second interactive control to a control page, determining a second predetermined task corresponding to the second interactive control; and adding a second interactive control to the control page, the second interactive control being used to invoke a digital assistant to perform the second predetermined task.
  • the third interaction request is text input received in the dialog page
  • determining the second predetermined task includes: determining the second predetermined task based on the text input.
  • process 300 further includes: in response to receiving a fourth interaction request for a dialog page, providing a second result in the dialog page, the second result being the result of the digital assistant performing a task specified by the fourth interaction request on an image.
  • process 300 further includes: providing a control in a dialog page to perform a follow-up task on a result in the dialog page, the result being the result of image processing by a digital assistant; and performing the follow-up task in response to receiving a fifth interaction request for the control.
  • process 300 further includes: in response to determining that the image is an image frame from a video segment, providing data associated with the video segment.
  • control page and the dialog page are floating pages displayed on top of the target page
  • process 300 further includes: in response to receiving a move request for moving either the control page or the dialog page, moving to the location of the page specified by the move request.
  • process 300 is implemented as a plug-in to a digital assistant, and process 300 is invoked in response to the activation of the plug-in.
  • FIG4 shows a block diagram of an apparatus 400 for processing images using a digital assistant according to some embodiments of this disclosure.
  • the apparatus 400 may be implemented as or included in a terminal device 110.
  • the various modules/components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.
  • the device 400 includes: a receiving module 410 configured to receive an image captured from a target page and send the image to a digital assistant; a determining module 420 configured to determine the processing intent for processing the image based on its content; and an acquisition module. 430 is configured to acquire the processing results associated with the image, which are determined by the digital assistant based on the processing intent.
  • the device 400 further includes: a dialogue page providing module configured to provide a dialogue page, the dialogue page being a conversational interactive page of a digital assistant, and the dialogue page including an image, a processing intent, and a processing result.
  • a dialogue page providing module configured to provide a dialogue page, the dialogue page being a conversational interactive page of a digital assistant, and the dialogue page including an image, a processing intent, and a processing result.
  • the processing intent includes multiple processing intents
  • the dialog page includes multiple prompts corresponding to the multiple processing intents respectively.
  • the acquisition module further includes: a selection-based acquisition module, configured to, in response to receiving a selection request for a target prompt among the multiple prompts, acquire a processing result based on the target processing intent corresponding to the target prompt among the multiple processing intents.
  • the acquisition module further includes: a context acquisition module configured to acquire context data of an image from a target page; and a context-based acquisition module configured to acquire processing results associated with the image based on the context data.
  • the acquisition module further includes: a control page providing module configured to provide a control page for invoking a digital assistant to process an image; and an assistant invoking module configured to invoke a digital assistant to process an image in response to receiving a first interaction request in the control page.
  • control page includes a first interactive control for performing a first predetermined task
  • page providing module 430 includes: a first result providing module configured to provide a first result in a dialog page in response to receiving a second interactive request for the first interactive control, the first result being the result of the digital assistant performing the first predetermined task on an image.
  • the apparatus 400 further includes: a second predetermined task determination module configured to determine a second predetermined task corresponding to the second interactive control in response to receiving a third interaction request for adding a second interactive control to a control page; and a control adding module configured to add a second interactive control to the control page, the second interactive control being used to invoke a digital assistant to perform the second predetermined task.
  • a second predetermined task determination module configured to determine a second predetermined task corresponding to the second interactive control in response to receiving a third interaction request for adding a second interactive control to a control page
  • a control adding module configured to add a second interactive control to the control page, the second interactive control being used to invoke a digital assistant to perform the second predetermined task.
  • the third interaction request is text input received in the dialog page
  • the second predetermined task determination module is specifically configured to determine the second predetermined task based on the text input.
  • the apparatus 400 further includes: a second result providing module configured to provide a second result in the dialog page in response to receiving a fourth interaction request for the dialog page, the second result being the result of the digital assistant performing a task specified by the fourth interaction request on an image.
  • a second result providing module configured to provide a second result in the dialog page in response to receiving a fourth interaction request for the dialog page, the second result being the result of the digital assistant performing a task specified by the fourth interaction request on an image.
  • the apparatus 400 further includes: a control providing module configured to provide a control in a dialog page for performing a follow-up task on a result in the dialog page, the result being an image processed by a digital assistant; and a follow-up task execution module configured to perform a follow-up task in response to receiving a fifth interaction request for the control.
  • a control providing module configured to provide a control in a dialog page for performing a follow-up task on a result in the dialog page, the result being an image processed by a digital assistant
  • a follow-up task execution module configured to perform a follow-up task in response to receiving a fifth interaction request for the control.
  • the apparatus 400 further includes a data providing module configured to provide data associated with the video segment in response to determining that the image is an image frame from the video segment.
  • control page and the dialog page are floating pages displayed on top of the target page
  • the apparatus 400 further includes: a page movement module configured to move to the position of the page specified by the movement request in response to receiving a movement request for moving either the control page or the dialog page.
  • device 400 is implemented as a plug-in to a digital assistant, and device 400 is invoked in response to the activation of the plug-in.
  • the units and/or modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof.
  • one or more units and/or modules can be implemented using software and/or firmware, such as machine-executable instructions stored on a storage medium.
  • some or all of the units and/or modules in device 400 can be implemented at least partially by one or more hardware logic components.
  • exemplary types of hardware logic components include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.
  • Figure 5 illustrates a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 500 shown in Figure 5 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement the terminal device 110 of FIG1, the server 130, and/or the device 400 of FIG4.
  • the electronic device 500 is in the form of a general-purpose computing device.
  • Components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560.
  • the processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 500.
  • Electronic device 500 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media.
  • Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof.
  • Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and/or data and accessible within electronic device 500.
  • Electronic device 500 may further include additional removable/non-removable, volatile/non-volatile storage media.
  • disk drives for reading or writing from removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading or writing from removable, non-volatile optical disks may be provided.
  • each drive may be connected to a bus (not shown) via one or more data media interfaces.
  • Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.
  • the communication unit 540 enables communication with other computing devices via a communication medium. Additionally, the components of the electronic device 500 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device... The 500 can operate in a networked environment using a logical connection to one or more other servers, network personal computers (PCs), or another network node.
  • PCs network personal computers
  • Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc.
  • Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc.
  • Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input/output (I/O) interface (not shown).
  • I/O input/output
  • a computer-readable storage medium that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above.
  • a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.
  • These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions/actions specified in one or more blocks of the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and/or other device to operate in a particular manner.
  • the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions/actions specified in one or more blocks of the flowchart and/or block diagram.
  • Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions/actions specified in one or more boxes of a flowchart and/or block diagram.
  • each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function.
  • the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.
  • each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

根据本公开的实施例，提供了利用数字助手处理图像的方法、装置、设备和存储介质。在该方法中，接收从目标页面中所截取的图像，向数字助手发送图像。基于图像的内容，确定用于处理图像的处理意图。由数字助手基于处理意图，获取与图像相关联的处理结果。以此方式，可以借助数字助手对截取到的图像进行处理，并在对话页面中呈现处理结果。这有助于提升处理图像的效率，降低用户获取信息的难度，提升用户获取信息的效率。

Description

利用数字助手处理图像的方法、装置、设备和存储介质 技术领域
本公开的示例实施例总体涉及图像处理领域，特别地涉及利用数字助手处理图像的方法、装置、设备和计算机可读存储介质。
背景技术
机器学习技术已经被广泛地用于多种技术领域。在现有技术中，数字助手被提供用于协助用户在不同应用、场景下的各类任务处理需求。数字助手通常具有智能对话和任务处理能力。数字助手可以响应于用户输入而被调用。例如，在与数字助手的交互过程中，用户输入交互消息，并且数字助手响应于用户输入来提供回复消息。
发明内容
在本公开的第一方面，提供了一种利用数字助手处理图像的方法。该方法包括：接收从目标页面中所截取的图像，向数字助手发送图像。基于图像的内容，确定用于处理图像的处理意图。获取与图像相关联的处理结果，处理结果是由数字助手基于处理意图来确定的。
在本公开的第二方面，提供了一种利用数字助手处理图像的装置。该装置包括：接收模块，被配置为接收从目标页面中所截取的图像，向数字助手发送图像；确定模块，被配置为基于图像的内容，确定用于处理图像的处理意图；以及获取模块，被配置为获取与图像相关联的处理结果，处理结果是由数字助手基于处理意图来确定的。
在本公开的第三方面，提供了一种电子设备。该设备包括至少一个处理单元；以及至少一个存储器，至少一个存储器被耦合到至少一个处理单元并且存储用于由至少一个处理单元执行的指令。指令在由至少一个处理单元执行时使设备执行本公开的第一方面的方法。
在本公开的第四方面，提供了一种计算机可读存储介质。该计算机可读存储介质上存储有计算机程序，计算机程序可由处理器执行以实现本公开的第一方面的方法。
在本公开的第五方面，提供了一种计算机程序产品，包括计算机程序，其中计算机程序在被处理器执行时实现本公开的第一方面的方法。
应当理解，本内容部分中所描述的内容并非旨在限定本公开的实施例的关键特征或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的描述而变得容易理解。
附图说明
结合附图并参考以下详细说明，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。在附图中，相同或相似的附图标记表示相同或相似的元素，其中：
图1示出了本公开的实施例能够在其中实现的示例环境的示意图；
图2A至图2C示出了根据本公开的一些实施例的示例页面的示意图；
图3示出了根据本公开的一些实施例的利用数字助手处理图像的过程的流程图；
图4示出了根据本公开的一些实施例的利用数字助手处理图像的装置的框图；以及
图5示出了其中可以实施本公开的一个或多个实施例的电子设备的框图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中示出了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反，提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本 公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
在本公开的实施例的描述中，术语“包括”及其类似用语应当理解为开放性包含，即“包括但不限于”。术语“基于”应当理解为“至少部分地基于”。术语“一个实施例”或“该实施例”应当理解为“至少一个实施例”。术语“一些实施例”应当理解为“至少一些实施例”。下文还可能包括其他明确的和隐含的定义。
在本文中，除非明确说明，“响应于A”执行一个步骤并不意味着在“A”之后立即执行该步骤，而是可以包括一个或多个中间步骤。
可以理解的是，本技术方案所涉及的数据(包括但不限于数据本身、数据的获得或使用)应当遵循相应法律法规及相关规定的要求。
可以理解的是，在使用本公开各实施例公开的技术方案之前，均应当根据相关法律法规通过适当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
例如，在响应于接收到用户的主动请求时，向用户发送提示信息，以明确地提示用户，其请求执行的操作将需要获得和使用到用户的个人信息，从而使得用户可以根据提示信息来自主地选择是否向执行本公开技术方案的操作的电子设备、应用程序、服务器或存储介质等软件或硬件提供个人信息。
作为一种可选的但非限制性的实现方式，响应于接收到用户的主动请求，向用户发送提示信息的方式，例如可以是弹窗的方式，弹窗中可以以文字的方式呈现提示信息。此外，弹窗中还可以承载供用户选择“同意”或“不同意”向电子设备提供个人信息的选择控件。
可以理解的是，上述通知和获得用户授权过程仅是示意性的，不对本公开的实现方式构成限定，其他满足相关法律法规的方式也可应用于本公开的实现方式中。
如本文中所使用的，术语“模型”可以从训练数据中学习到相应的输入与输出之间的关联关系，从而在训练完成后可以针对给定的输入，生成对应的输出。模型的生成可以基于机器学习技术。深度学习 是一种机器学习算法，通过使用多层处理单元来处理输入和提供相应输出。神经网络模型是基于深度学习的模型的一个示例。在本文中，“模型”也可以被称为“机器学习模型”、“学习模型”、“机器学习网络”或“学习网络”，这些术语在本文中可互换地使用。
“神经网络”是一种基于深度学习的机器学习网络。神经网络能够处理输入并且提供相应输出，其通常包括输入层和输出层以及在输入层与输出层之间的一个或多个隐藏层。在深度学习应用中使用的神经网络通常包括许多隐藏层，从而增加网络的深度。神经网络的各个层按顺序相连，从而前一层的输出被提供作为后一层的输入，其中输入层接收神经网络的输入，而输出层的输出作为神经网络的最终输出。神经网络的每个层包括一个或多个节点(也称为处理节点或神经元)，每个节点处理来自上一层的输入。
通常，机器学习大致可以包括三个阶段，即训练阶段、测试阶段和应用阶段(也称为推理阶段)。在训练阶段，给定的模型可以使用大量的训练数据进行训练，不断迭代更新参数值，直到模型能够从训练数据中获得一致的满足预期目标的推理。通过训练，模型可以被认为能够从训练数据中学习从输入到输出之间的关联(也称为输入到输出的映射)。训练后的模型的参数值被确定。在测试阶段，将测试输入应用到训练后的模型，以测试模型是否能够提供正确的输出，从而确定模型的性能。测试阶段有时候可以被融合在训练阶段。在应用或推理阶段，已训练的模型可以被用于基于训练得到的参数值，对实际的模型输入进行处理，确定对应的模型输出。
图1示出了本公开的实施例能够在其中实现的示例环境100的示意图。在该示例环境100中，数字助手120可以被安装在终端设备110中(也可以称之为终端设备110中部署有数字助手120)。用户140可以经由终端设备110和/或终端设备110的附接设备来与数字助手120进行交互。
在一些实施例中，数字助手120可以被包括在或者被实现为目标应用，目标应用为终端设备110中已安装的应用。数字助手120可以 被配置为具有智能对话能力和在目标应用内的任务处理能力。数字助手120被提供用于协助用户在不同应用、场景下的各类任务处理需求。在与数字助手120的交互过程中，用户输入交互消息，并且数字助手120响应于用户输入来提供回复消息。交互消息可以包括多模态形式的消息，诸如文本消息(例如，自然语言文本)、语音消息、图像消息、视频消息，等等。通常，数字助手120能够支持用户以自然语言方式输入提问，并基于对自然语言输入的理解和逻辑推理能力来执行任务和提供回复。在一些实施例中，目标应用可以基于用户140的操作而向用户140呈现相应的页面150，以向用户140输出和/或从用户140接收信息。
在一些实施例中，目标应用的至少部分功能的实现，和/或目标应用中的数字助手120的至少部分功能的实现可以基于目标模型来实现。在一些实施例中，目标模型可以是机器学习模型、深度学习模型、学习模型、神经网络等。在一些实施例中，模型可以基于语言模型(LM)。语言模型通过从大量语料中学习，能够具备问答能力。目标模型也可以基于其他适当的模型。
在一些实施例中，终端设备110与服务器130通信，以实现对目标应用和/或数字助手120的服务的供应。终端设备110可以是任意类型的移动终端、固定终端或便携式终端，包括移动手机、台式计算机、膝上型计算机、笔记本计算机、上网本计算机、平板计算机、媒体计算机、多媒体平板、个人通信系统(PCS)设备、个人导航设备、个人数字助理(PDA)、音频/视频播放器、数码相机/摄像机、定位设备、电视接收器、无线电广播接收器、电子书设备、游戏设备或者前述各项的任意组合，包括这些设备的配件和外设或者其任意组合。在一些实施例中，终端设备110也能够支持任意类型的针对用户的接口(诸如“可佩戴”电路等)。服务器130可以是能够提供计算能力的各种类型的计算系统/服务器，包括但不限于大型机、边缘计算节点、云环境中的计算设备，等等。
应当理解，仅出于示例性的目的描述环境100中各个元素的结构 和功能，而不暗示对于本公开的范围的任何限制。
在快节奏的数字时代，用户被各种信息和媒体内容包围。海量信息的消费和处理成为了用户日常生活中的一大挑战。例如。人们常常会分享有趣的内容或查找某个物品的相关信息。又例如，人们在工作中需要从包括复杂数据的文档中快速提取关键信息，与团队即时分享，并进行深入分析。又例如，学生需要从在线资源中整理笔记，等等。用户期望可以提供一个不仅能够帮助他们处理信息，还能够适应他们多变的需求和个性化的习惯的工具。
有鉴于此，根据本公开的实施例，提供了一种利用数字助手处理图像的改进方案。根据该方案，接收从目标页面中所截取的图像，向数字助手发送图像。基于图像的内容，确定用于处理图像的处理意图。获取与所述图像相关联的处理结果，所述处理结果是由所述数字助手基于所述处理意图来确定的。由此，可以借助数字助手来基于处理意图，对截取到的图像进行处理，并在对话页面中呈现处理结果。这有助于提升处理图像的效率，降低用户获取信息的难度，提升用户获取信息的效率。
以下将参考附图的示例来详细描述本公开的一些示例实施例。在一些实施例中，本公开所提供的利用数字助手处理图像的方法可以作为数字助手的插件而被实现，并且方法响应于插件的激活而被调用。例如，数字助手可以提供指示是否开启方法的开关。可以仅在该开关处于启用状态的情况下，执行利用数字助手处理图像的方法。
图2A至图2C示出了根据本公开的一些实施例的示例页面200A至示例页面200C(也可以简称为示例200A至示例200C)的示意图。应当理解，附图中示出的页面仅仅是示例，实际可以存在各种页面设计。页面中的各个图形元素可以具有不同的布置和不同的视觉表示，其中的一个或多个元素可以被省略或被替换，并且还可以存在一个或多个其他元素。本公开的实施例在此方面不受限制。
示例200A至示例200C所示的页面可以被呈现在终端设备110处。为便于讨论，将参考图1的环境100来描述示例200A至示例200C。 需要注意的是，在本文中，参考终端设备110所描述的一些操作可能需要服务器130的协助完成。此外，终端设备110所执行的操作，具体可以是安装在终端设备110上的相关应用和/或数字助手120执行的。
在本公开的实施例中，终端设备110接收从目标页面中所截取的图像。目标页面可以是允许从中截取内容的任意适当页面，包括但不限于文档页面、网页、视频页面、媒体项浏览页面、消息页面，等等。如图2A和图2B所示，在示例200A中，终端设备110可以接收从目标页面210截取的图像201，在示例200B中，终端设备110可以接收从目标页面210截取的图像202。在此，该图像201可以是通过调用操作系统的截图功能所截取的图像，和/或调用其他应用的截图功能所截取的图像。
在本公开的实施例中，终端设备110向数字助手120发送图像，以便由数字助手120处理图像(这个过程也可以称之为调用数字助手120来处理图像)。在一些实施例中，终端设备110可以提供调用控件(例如，图2B中所示的调用页面220中的各个控件)。调用控件可以响应于获取到图像(也即上述从目标页面中所截取的图像)而被提供，也可以默认一直呈现，例如以浮动方式显示在目标页面上层。终端设备110例如可以响应于接收到针对调用控件的用户操作(例如用户针对调用控件的触发操作)，向数字助手120发送图像，以便由数字助手120处理图像。
在一些实施例中，终端设备110还可以提供用于调用数字助手120来处理图像的控件页面，并响应于在控件页面中接收到第一交互请求，调用数字助手120来处理图像。控件页面例如可以为在目标页面上层显示的浮动页面。如图2A和图2B所示，终端设备110可以提供控件页面220，控件页面220为在目标页面210上层显示的浮动页面。控件页面220例如可以包括多个交互控件，多个交互控件包括具有调用数字助手功能的至少一个交互控件(例如示例200A中的交互控件222、交互控件223和交互控件224，示例200B中的交互控件225和交互控件226)。终端设备110可以响应于接收到针对目标交互控件 (具有调用数字助手功能的任一交互控件)的第一交互请求(例如针对目标操作控件的用户操作)，调用数字助手120来处理图像。
在一些实施例中，可以基于图像的内容，确定用于处理图像的处理意图。假设图像包括文本内容，则可以确定处理意图为：自动识别、自动翻译、自动搜索等等。具体地，可以基于文本识别来确定细化的处理意图，假设图像包括姓名和联系方式等，则可以确定处理意图为添加联系人，假设图像包括时间、地点等信息，则可以确定处理意图为创建日程，等等。此时，可以调用数字助手120来确定处理结果。
又例如，可以识别图像中的对象，进而确定相应的处理意图。假设图像包括某个物品，则可以确定处理意图为：查询详细信息、购买，等等。继而，可以调用数字助手120来提供相应的查询结果、或者购买链接，等等。
在一些实施例中，可以提供对话页面，对话页面是数字助手的对话式交互页面，并且对话页面包括图像、处理意图、以及处理结果。在本公开的实施例中，终端设备110可以响应于数字助手120被调用来提供对话页面，对话页面是数字助手的对话式交互页面，并且对话页面包括图像。如图2C所示，终端设备110例如可以在调用数字助手120处理图像的情况下，呈现对话页面230。对话页面230为用户(例如用户140)和数字助手120之间的对话页面。对话页面230例如同样可以为在目标页面210上层显示的浮动页面。
对话页面230可以包括图像233、处理意图(例如，会话消息234表示处理意图为查找相关网页)、以及处理结果(例如，会话消息235表示来自数字助手的应答)。在一些实施例中，可以由用户将图像拷贝至对话页面中；备选地和/或附加地，在此图像233可以被自动地呈现在对话页面中，以此方式，可以进一步简化用户操作并且提高获取信息的效率。
在一些实施例中，针对某个图像而言，可以从多方面确定多个处理意图。此时，处理意图包括多个处理意图，并且在获取与图像相关联的处理结果的过程中，对话页面包括分别对应于多个处理意图的多 个提示。此时，用户可以从多个提示中选择期望的处理意图，由此指定数字助手的任务。进一步，响应于接收到针对多个提示中的目标提示的选择请求，基于多个处理意图中的对应于目标提示的目标处理意图，获取处理结果。
继续上文的示例，假设图像包括文本内容，则可以确定多个处理意图：自动识别、自动翻译、自动搜索等等。可以分别提供对应于上述多个处理意图的多个提示，进而基于用户选择的提示来处理图像。假设用户选择“自动识别”，则处理结果可以表示文本识别的结果。假设用户选择“自动翻译”，则可以基于用户使用数字助手的历史交互中所涉及的语言来作为目标语言，备选地和/或附加地，可以进一步要求用户指定目标语言，提供相应的翻译结果。假设用户选择“自动搜索”，则可以在指定范围内搜索对应于该文本内容的结果。此时，该结果可以涉及多种媒体格式，例如，文本、图像、音频、视频，等等。
在一些实施例中，获取与图像相关联的处理结果可以进一步包括：从目标页面中获取图像的上下文数据，进而基于上下文数据，获取与图像相关联的处理结果。假设用户从论文中截取了附图，此时可以将论文中的与该附图相关联的文本内容作为图像的上下文数据。进一步，可以利用上下文数据来分析附图表示的具体含义。假设附图包括多个参数的多个曲线，则数字助手可以利用上下文数据中的各个参数的具体定义，来解释各个曲线的含义、多个参数之间的关联关系，等等。又例如，假设用户从新闻网页中截取了照片，则数字助手可以将新闻网页中的文字报道作为上下文数据，并且解释照片中的各个人物的身份，等等。
在一些实施例中，终端设备110可以响应接收到用于移动控件页面和对话页面中的任一项的移动请求，移动至由移动请求指定的页面的位置。终端设备110例如可以响应于接收到针对控件页面和/或对话页面的拖动操作，确定接收到针对控件页面和/或对话页面的移动请求。终端设备110例如可以将拖动操作所结束的位置确定为移动请求所指 定的位置。备选地和/或附加地，对话页面230还可以在固定位置处被显示。
在一些实施例中，终端设备110可以响应于确定对话页面230被呈现，或者，响应于确定要呈现对话页面230，而不再呈现(也即，隐藏)控件页面220。在一些实施例中，终端设备110可以在目标页面210上层呈现对话页面230的同时，保持控件页面220的呈现(也即控件页面220和对话页面230可以一并呈现)。对话页面230包括取消控件232。终端设备110可以响应于接收到针对取消控件232的用户操作，取消呈现对话页面230。对话页面230中呈现有图像233，图像233例如可以为终端设备110先前从目标页面210中所截取的图像。图像233可以以来自用户的会话消息的形式呈现在对话页面230中。
在一些实施例中，控件页面包括用于执行第一预定任务的第一交互控件。返回2A和图2B，在示例200A中，控件页面220可以包括用于执行详细说明任务的交互控件222和执行网页搜索任务的交互控件223。在示例200B中，控件页面220可以包括用于执行查看词条任务的交互控件225和用于执行搜索相关内容任务的交互控件226。
终端设备110可以响应于接收到针对第一交互控件的第二交互请求，在对话页面中提供第一结果，第一结果是由数字助手120针对图像执行第一预定任务的结果。第一预定任务可以是预先被指定的任务。示例性地，终端设备110可以响应于接收到针对交互控件222的第二交互请求，来执行详细说明任务。如示例200C所示的对话页面230，可以提供由数字助手120执行对图像中的内容进行详细说明任务所得到的第一结果，例如，呈现从图像中提取的文本和/或其他信息，等等。
在一些实施例中，终端设备110可以接收用于向控件页面中添加第二交互控件的第三交互请求。在一些实施例中，终端设备110可以响应于接收到针对添加控件的用户操作，确定接收到用于向控件页面中添加第二交互控件的第三交互请求。示例性地，控件页面可以包括第一添加控件(例如图2A和图2B所示的控件页面220中的添加控 件221)，终端设备110可以响应于接收到针对第一添加控件的用户操作，确定接收到用于向控件页面中添加第二交互控件的第三交互请求。以此方式，用户可以按照自身需求来添加个性化的控件。
示例性地，对话页面可以包括设置控件(例如图2C中对话页面230中的设置控件231)，终端设备110例如可以响应于接收到针对设置控件的用户操作，呈现设置页面。设置页面例如可以包括第二添加控件。备选地或者附加地，对话页面也可以直接包括第二添加控件。终端设备110可以响应于接收到针对第二添加控件的用户操作，确定接收到用于向控件页面中添加第二交互控件的第三交互请求。
在一些实施例中，终端设备110可以响应于接收到针对第一添加控件的用户操作，确定接收到用于向控件页面中添加第二交互控件的第三交互请求。终端设备110还可以响应于接收到针对第一添加控件的用户操作，呈现用于指定第二预定任务的设置页面。终端设备110可以经由该页面接收用户输入，并基于用户输入确定第二预定任务。例如，预定任务例如可以包括但不限于提取图像中的文本、识别图像中的对象，等等。以此方式，可以便于用户向控件页面中添加控件，从而以更为简单的方式执行预定任务。
在一些实施例中，第三交互请求还可以是在对话页面中接收的文本输入，终端设备110可以基于文本输入来确定第二预定任务。例如，终端设备110可以响应于在对话页面中接收到例如“在控件页面添加XXXX任务对应的控件”这样的文本，确定第二预定任务为XXXX任务。
终端设备110进而可以响应于接收到用于向控件页面中添加第二交互控件的第三交互请求，确定对应于第二交互控件的第二预定任务，并向控件页面中添加用于调用数字助手来执行第二预定任务的第二交互控件。示例性地，如果终端设备110确定第二预定任务为识别图像中的公式这一任务，则终端设备110可以向控件页面中添加用于调用数字助手来执行识别图像中的公式这一任务的第二交互控件。
由此，允许用户自行定义控件页面中用于调用数字助手的控件， 可以提高数字助手执行相应任务的效率。
在一些实施例中，控件页面中用于调用数字助手的至少一个交互控件可以包括仅用于调用数字助手的交互控件(例如图2A中的交互控件224)。终端设备110可以响应于该交互控件被触发而呈现对话页面。进一步，终端设备110可以经由对话页面接收用户输入。终端设备110可以将接收到的用户输入确定为第四交互请求。终端设备110可以响应于接收到针对对话页面的第四交互请求，在对话页面中提供第二结果，第二结果是由数字助手针对图像执行由第四交互请求指定的任务的结果。
如图2C所示，终端设备110可以经由输入框236接收用户输入，并将用户输入确定为第四交互请求。终端设备110可以以来自用户的会话消息(例如图中所示的会话消息234)的形式呈现接收到的用户输入。数字助手120可以执行该第四交互请求所指定的任务。终端设备110可以呈现数字助手120执行第四交互请求所指定的任务所得到的结果，例如以来自数字助手120的会话消息(也即图中所示的会话消息235)的形式呈现在对话页面230中。
在一些实施例中，终端设备110还可以在对话页面中提供针对对话页面中的结果(也即由数字助手来处理图像的结果)执行后续任务的控件。执行后续任务的控件例如可以包括编辑控件、发送控件、收藏控件等任意适当的控件。终端设备110可以响应于接收到针对控件的第五交互请求，执行后续任务。例如，终端设备110可以响应于接收到针对收藏控件的用户操作，将结果添加至用户的收藏夹。
需要注意的是，在一些实施例中，在确定图像为来自视频片段的图像帧的情况下，终端设备110还可以提供与视频片段相关联的数据。返回图2B，如果目标页面210为电影的播放页面，图像202为包括电影中的某个角色的图像，则终端设备110可以指示数字助手120确定图像202是哪一部电影中的图像以及该角色是电影中的哪个角色，并在对话页面中提供与该电影相关联的电影介绍、角色简介、电影评价等信息。
在教育辅助场景中，当用户在阅读电子教材或查看教学视频截图时，终端设备110能够基于截图中的内容，在对话页面为用户显示相关的历史人物资料、科学理论解释或数学问题的解法。这可以提高用户获取知识的效率。
在购物推荐场景中，在用户浏览商品图片或相关广告截图时，终端设备110可以基于截图中的内容，在对话页面中提供商品的比价信息，并基于用户的购物历史和偏好，推荐个性化的优惠券、相关商品组合或替代品，提升购物便利性和效率。
在健康咨询场景中，在用户在查看关于健康或饮食的内容截图时，终端设备110可以基于截图中的内容，在对话页面中提供营养成分分析、健康建议或食谱推荐。对于运动相关的截图，则能够在对话页面中推荐相应的锻炼方法或提供接近用户所在地的运动场所推荐。
在旅游规划场景中，当用户看到旅游地点的照片或截图时，终端设备110可以基于截图中的内容，在对话页面中提供图像中的地点的介绍，并提供详细的旅游指南、实时天气、当地文化背景知识、旅行社推荐和预算规划等。
在多语言学习场景中，终端设备110可以基于截图中的内容，在对话页面中提供即时翻译。同时，也能提供该语言的发音、语法解析、使用场景和练习对话，帮助用户提高语言学习效率。
在家居装修场景在，用户上传家居装修的图片或设计图截图时，终端设备110可以基于截图中的内容，在对话页面中提供装修建议,如色彩搭配、家具选择和空间利用方案，甚至能推荐本地的装修服务商科和相关优惠信息。
在影视内容扩展场景中，在用户观看电影或剧集的截图时，终端设备110可以基于截图中的内容，在对话页面中提供角色分析、剧情讨论、幕后花絮和相关文学作品或历史事件的深度资料，以及类似影视作品的推荐，可以增强影视观看后的信息获取效率。
综上所述，根据本公开的实施例，可以自动地将截图发送至数字助手，进而可以借助数字助手对截取到的图像进行处理，并在对话页 面中呈现处理结果。这有助于提升处理图像的效率，降低用户获取信息的难度，提升用户获取信息的效率。
上文已经描述了利用数字助手处理图像的各个步骤的具体细节，提供了一种利用数字助手处理图像的方法。图3示出了根据本公开的一些实施例的利用数字助手处理图像的过程300的流程图。过程300可以被实现在终端设备110处。下面参考图1描述过程300。
在框310，接收从目标页面中所截取的图像，向数字助手发送图像。
在框320，基于图像的内容，确定用于处理图像的处理意图。
在框330，获取与图像相关联的处理结果，处理结果是由数字助手基于处理意图来确定的。
在一些实施例中，过程300进一步包括：提供对话页面，对话页面是数字助手的对话式交互页面，并且对话页面包括图像、处理意图、以及处理结果。
在一些实施例中，处理意图包括多个处理意图，并且对话页面包括分别对应于多个处理意图的多个提示，获取与图像相关联的处理结果进一步包括：响应于接收到针对多个提示中的目标提示的选择请求，基于多个处理意图中的对应于目标提示的目标处理意图，获取处理结果。
在一些实施例中，获取与图像相关联的处理结果进一步包括：从目标页面中获取图像的上下文数据；以及基于上下文数据，获取与图像相关联的处理结果。
在一些实施例中，获取与图像相关联的处理结果进一步包括：提供用于调用数字助手来处理图像的控件页面；以及响应于在控件页面中接收到第一交互请求，调用数字助手来处理图像。
在一些实施例中，控件页面包括用于执行第一预定任务的第一交互控件，其中提供对话页面包括：响应于接收到针对第一交互控件的第二交互请求，在对话页面中提供第一结果，第一结果是由数字助手针对图像执行第一预定任务的结果。
在一些实施例中，过程300进一步包括：响应于接收到用于向控件页面中添加第二交互控件的第三交互请求，确定对应于第二交互控件的第二预定任务；以及向控件页面中添加第二交互控件，第二交互控件用于调用数字助手来执行第二预定任务。
在一些实施例中，第三交互请求是在对话页面中接收的文本输入，以及确定第二预定任务包括：基于文本输入来第二预定任务确定。
在一些实施例中，过程300进一步包括：响应于接收到针对对话页面的第四交互请求，在对话页面中提供第二结果，第二结果是由数字助手针对图像执行由第四交互请求指定的任务的结果。
在一些实施例中，过程300进一步包括：在对话页面中提供针对对话页面中的结果执行后续任务的控件，结果是由数字助手来处理图像的结果；以及响应于接收到针对控件的第五交互请求，执行后续任务。
在一些实施例中，过程300进一步包括：响应于确定图像为来自视频片段的图像帧，提供与视频片段相关联的数据。
在一些实施例中，控件页面和对话页面是在目标页面上层显示的浮动页面，并且过程300进一步包括：响应接收到用于移动控件页面和对话页面中的任一项的移动请求，移动至由移动请求指定的页面的位置。
在一些实施例中，过程300作为数字助手的插件而被实现，并且过程300响应于插件的激活而被调用。
根据本公开的一些实施例，还提供了一种利用数字助手处理图像的装置。图4示出了根据本公开的一些实施例的利用数字助手处理图像的装置400的框图。装置400可以被实现为或者被包括在终端设备110中。装置400中的各个模块/组件可以由硬件、软件、固件或者它们的任意组合来实现。
如图4所示，装置400包括：接收模块410，被配置为接收从目标页面中所截取的图像，向数字助手发送图像；确定模块420，被配置为基于图像的内容，确定用于处理图像的处理意图；以及获取模块 430，被配置为获取与图像相关联的处理结果，处理结果是由数字助手基于处理意图来确定的。
在一些实施例中，装置400进一步包括：对话页面提供模块，被配置为提供对话页面，对话页面是数字助手的对话式交互页面，并且对话页面包括图像、处理意图、以及处理结果。
在一些实施例中，处理意图包括多个处理意图，并且对话页面包括分别对应于多个处理意图的多个提示，以及获取模块进一步包括：基于选择的获取模块，被配置为响应于接收到针对多个提示中的目标提示的选择请求，基于多个处理意图中的对应于目标提示的目标处理意图，获取处理结果。
在一些实施例中，获取模块进一步包括：上下文获取模块，被配置为从目标页面中获取图像的上下文数据；以及基于上下文的获取模块，被配置为基于上下文数据，获取与图像相关联的处理结果。
在一些实施例中，获取模块进一步包括：控件页面提供模块，被配置为提供用于调用数字助手来处理图像的控件页面；以及助手调用模块，被配置为响应于在控件页面中接收到第一交互请求，调用数字助手来处理图像。
在一些实施例中，控件页面包括用于执行第一预定任务的第一交互控件，其中页面提供模块430包括：第一结果提供模块，被配置为响应于接收到针对第一交互控件的第二交互请求，在对话页面中提供第一结果，第一结果是由数字助手针对图像执行第一预定任务的结果。
在一些实施例中，装置400进一步包括：第二预定任务确定模块，被配置为响应于接收到用于向控件页面中添加第二交互控件的第三交互请求，确定对应于第二交互控件的第二预定任务；以及控件添加模块，被配置为向控件页面中添加第二交互控件，第二交互控件用于调用数字助手来执行第二预定任务。
在一些实施例中，第三交互请求是在对话页面中接收的文本输入，以及第二预定任务确定模块具体被配置为：基于文本输入来确定第二预定任务。
在一些实施例中，装置400进一步包括：第二结果提供模块，被配置为响应于接收到针对对话页面的第四交互请求，在对话页面中提供第二结果，第二结果是由数字助手针对图像执行由第四交互请求指定的任务的结果。
在一些实施例中，装置400进一步包括：控件提供模块，被配置为在对话页面中提供针对对话页面中的结果执行后续任务的控件，结果是由数字助手来处理图像的结果；以及后续任务执行模块，被配置为响应于接收到针对控件的第五交互请求，执行后续任务。
在一些实施例中，装置400进一步包括：数据提供模块，被配置为响应于确定图像为来自视频片段的图像帧，提供与视频片段相关联的数据。
在一些实施例中，控件页面和对话页面是在目标页面上层显示的浮动页面，并且装置400进一步包括：页面移动模块，被配置为响应接收到用于移动控件页面和对话页面中的任一项的移动请求，移动至由移动请求指定的页面的位置。
在一些实施例中，装置400作为数字助手的插件而被实现，并且装置400响应于插件的激活而被调用。
装置400中所包括的单元和/或模块可以利用各种方式来实现，包括软件、硬件、固件或其任意组合。在一些实施例中，一个或多个单元和/或模块可以使用软件和/或固件来实现，例如存储在存储介质上的机器可执行指令。除了机器可执行指令之外或者作为替代，装置400中的部分或者全部单元和/或模块可以至少部分地由一个或多个硬件逻辑组件来实现。作为示例而非限制，可以使用的示范类型的硬件逻辑组件包括现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准品(ASSP)、片上系统(SOC)、复杂可编程逻辑器件(CPLD)，等等。
图5示出了其中可以实施本公开的一个或多个实施例的电子设备500的框图。应当理解，图5所示出的电子设备500仅仅是示例性的，而不应当构成对本文所描述的实施例的功能和范围的任何限制。图5 所示出的电子设备500可以用于实现图1的终端设备110，服务器130，和/或，图4的装置400。
如图5所示，电子设备500是通用计算设备的形式。电子设备500的组件可以包括但不限于一个或多个处理器或处理单元510、存储器520、存储设备530、一个或多个通信单元540、一个或多个输入设备550以及一个或多个输出设备560。处理单元510可以是实际或虚拟处理器并且能够根据存储器520中存储的程序来执行各种处理。在多处理器系统中，多个处理单元并行执行计算机可执行指令，以提高电子设备500的并行处理能力。
电子设备500通常包括多个计算机存储介质。这样的介质可以是电子设备500可访问的任何可以获得的介质，包括但不限于易失性和非易失性介质、可拆卸和不可拆卸介质。存储器520可以是易失性存储器(例如寄存器、高速缓存、随机访问存储器(RAM))、非易失性存储器(例如，只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、闪存)或它们的某种组合。存储设备530可以是可拆卸或不可拆卸的介质，并且可以包括机器可读介质，诸如闪存驱动、磁盘或者任何其他介质，其可以能够用于存储信息和/或数据并且可以在电子设备500内被访问。
电子设备500可以进一步包括另外的可拆卸/不可拆卸、易失性/非易失性存储介质。尽管未在图5中示出，可以提供用于从可拆卸、非易失性磁盘(例如“软盘”)进行读取或写入的磁盘驱动和用于从可拆卸、非易失性光盘进行读取或写入的光盘驱动。在这些情况中，每个驱动可以由一个或多个数据介质接口被连接至总线(未示出)。存储器520可以包括计算机程序产品525，其具有一个或多个程序模块，这些程序模块被配置为执行本公开的各种实现方式的各种方法或动作。
通信单元540实现通过通信介质与其他计算设备进行通信。附加地，电子设备500的组件的功能可以以单个计算集群或多个计算机器来实现，这些计算机器能够通过通信连接进行通信。因此，电子设备 500可以使用与一个或多个其他服务器、网络个人计算机(PC)或者另一个网络节点的逻辑连接来在联网环境中进行操作。
输入设备550可以是一个或多个输入设备，例如鼠标、键盘、追踪球等。输出设备560可以是一个或多个输出设备，例如显示器、扬声器、打印机等。电子设备500还可以根据需要通过通信单元540与一个或多个外部设备(未示出)进行通信，外部设备诸如存储设备、显示设备等，与一个或多个使得用户与电子设备500交互的设备进行通信，或者与使得电子设备500与一个或多个其他计算设备通信的任何设备(例如，网卡、调制解调器等)进行通信。这样的通信可以经由输入/输出(I/O)接口(未示出)来执行。
根据本公开的示例性实现方式，提供了一种计算机可读存储介质，其上存储有计算机可执行指令，其中计算机可执行指令被处理器执行以实现上文描述的方法。根据本公开的示例性实现方式，还提供了一种计算机程序产品，计算机程序产品被有形地存储在非瞬态计算机可读介质上并且包括计算机可执行指令，而计算机可执行指令被处理器执行以实现上文描述的方法。
这里参照根据本公开实现的方法、装置、设备和计算机程序产品的流程图和/或框图描述了本公开的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理单元，从而生产出一种机器，使得这些指令在通过计算机或其他可编程数据处理装置的处理单元执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
可以把计算机可读程序指令加载到计算机、其他可编程数据处理装置、或其他设备上，使得在计算机、其他可编程数据处理装置或其他设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其他可编程数据处理装置、或其他设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实现的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
以上已经描述了本公开的各实现，上述说明是示例性的，并非穷尽性的，并且也不限于所公开的各实现。在不偏离所说明的各实现的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实现的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其他普通技术人员能理解本文公开的各个实现方式。

Claims (17)

  1. 一种利用数字助手处理图像的方法，包括：
    接收从目标页面中所截取的图像，向所述数字助手发送所述图像；
    基于所述图像的内容，确定用于处理所述图像的处理意图；以及
    获取与所述图像相关联的处理结果，所述处理结果是由所述数字助手基于所述处理意图来确定的。
  2. 根据权利要求1所述的方法，进一步包括：提供对话页面，所述对话页面是所述数字助手的对话式交互页面，并且所述对话页面包括所述图像、所述处理意图、以及所述处理结果。
  3. 根据权利要求1所述的方法，其中所述处理意图包括多个处理意图，并且所述对话页面包括分别对应于所述多个处理意图的多个提示，获取与所述图像相关联的处理结果进一步包括：
    响应于接收到针对所述多个提示中的目标提示的选择请求，基于所述多个处理意图中的对应于所述目标提示的目标处理意图，获取所述处理结果。
  4. 根据权利要求1所述的方法，其中获取与所述图像相关联的处理结果进一步包括：
    从所述目标页面中获取所述图像的上下文数据；以及
    基于所述上下文数据，获取与所述图像相关联的处理结果。
  5. 根据权利要求1所述的方法，其中获取与所述图像相关联的处理结果进一步包括：
    提供用于调用所述数字助手来处理所述图像的控件页面；以及
    响应于在所述控件页面中接收到第一交互请求，调用所述数字助手来处理所述图像。
  6. 根据权利要求5所述的方法，其中所述控件页面包括用于执行第一预定任务的第一交互控件，其中提供所述对话页面包括：响应于接收到针对所述第一交互控件的第二交互请求，在所述对话页面中提供第一结果，所述第一结果是由所述数字助手针对所述图像执行所 述第一预定任务的结果。
  7. 根据权利要求6所述的方法，进一步包括：响应于接收到用于向所述控件页面中添加第二交互控件的第三交互请求，
    确定对应于所述第二交互控件的第二预定任务；以及
    向所述控件页面中添加第二交互控件，所述第二交互控件用于调用所述数字助手来执行所述第二预定任务。
  8. 根据权利要求7所述的方法，其中所述第三交互请求是在所述对话页面中接收的文本输入，以及确定所述第二预定任务包括：基于所述文本输入来确定所述第二预定任务。
  9. 根据权利要求2所述的方法，进一步包括：
    响应于接收到针对所述对话页面的第四交互请求，在所述对话页面中提供第二结果，所述第二结果是由所述数字助手针对所述图像执行由所述第四交互请求指定的任务的结果。
  10. 根据权利要求2所述的方法，进一步包括：
    在所述对话页面中提供针对所述对话页面中的结果执行后续任务的控件，所述结果是由所述数字助手来处理所述图像的结果；以及
    响应于接收到针对所述控件的第五交互请求，执行所述后续任务。
  11. 根据权利要求2所述的方法，进一步包括：响应于确定所述图像为来自视频片段的图像帧，提供与所述视频片段相关联的数据。
  12. 根据权利要求5所述的方法，其中所述控件页面和所述对话页面是在所述目标页面上层显示的浮动页面，并且所述方法进一步包括：响应接收到用于移动所述控件页面和所述对话页面中的任一项的移动请求，移动至由所述移动请求指定的页面的位置。
  13. 根据权利要求1所述的方法，其中所述方法作为所述数字助手的插件而被实现，并且所述方法响应于所述插件的激活而被调用。
  14. 一种利用数字助手处理图像的装置，包括：
    接收模块，被配置为接收从目标页面中所截取的图像，向所述数字助手发送所述图像；
    确定模块，被配置为基于所述图像的内容，确定用于处理所述图 像的处理意图；以及
    获取模块，被配置为获取与所述图像相关联的处理结果，所述处理结果是由所述数字助手基于所述处理意图来确定的。
  15. 一种电子设备，包括：
    至少一个处理单元；以及
    至少一个存储器，所述至少一个存储器被耦合到所述至少一个处理单元并且存储用于由所述至少一个处理单元执行的指令，所述指令在由所述至少一个处理单元执行时使所述电子设备执行根据权利要求1至13中任一项所述的方法。
  16. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序可由处理器执行以实现根据权利要求1至13中任一项所述的方法。
  17. 一种计算机程序产品，包括计算机程序，其中所述计算机程序在被处理器执行时实现根据权利要求1至13中任一项所述的方法。
PCT/CN2024/095791 2024-05-28 2024-05-28 利用数字助手处理图像的方法、装置、设备和存储介质 Pending WO2025245697A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2024/095791 WO2025245697A1 (zh) 2024-05-28 2024-05-28 利用数字助手处理图像的方法、装置、设备和存储介质
CN202480002189.XA CN119156648A (zh) 2024-05-28 2024-05-28 利用数字助手处理图像的方法、装置、设备和存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2024/095791 WO2025245697A1 (zh) 2024-05-28 2024-05-28 利用数字助手处理图像的方法、装置、设备和存储介质

Publications (1)

Publication Number Publication Date
WO2025245697A1 true WO2025245697A1 (zh) 2025-12-04

Family

ID=93808798

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/095791 Pending WO2025245697A1 (zh) 2024-05-28 2024-05-28 利用数字助手处理图像的方法、装置、设备和存储介质

Country Status (2)

Country Link
CN (1) CN119156648A (zh)
WO (1) WO2025245697A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210072880A1 (en) * 2019-09-06 2021-03-11 Samsung Electronics Co., Ltd. Method and electronic device for generating at least one data item from screenshot
CN117008778A (zh) * 2023-07-24 2023-11-07 浙江天猫技术有限公司 页面信息处理方法、装置、设备及存储介质
CN117271872A (zh) * 2022-06-15 2023-12-22 北京有竹居网络技术有限公司 推荐方法、装置、设备、存储介质及计算机程序产品
CN117828065A (zh) * 2024-03-06 2024-04-05 深圳荣灿大数据技术有限公司 一种数字人客服方法、系统、装置及储存介质
CN117972160A (zh) * 2024-02-21 2024-05-03 暗物智能科技(广州)有限公司 一种多模态信息处理方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210072880A1 (en) * 2019-09-06 2021-03-11 Samsung Electronics Co., Ltd. Method and electronic device for generating at least one data item from screenshot
CN117271872A (zh) * 2022-06-15 2023-12-22 北京有竹居网络技术有限公司 推荐方法、装置、设备、存储介质及计算机程序产品
CN117008778A (zh) * 2023-07-24 2023-11-07 浙江天猫技术有限公司 页面信息处理方法、装置、设备及存储介质
CN117972160A (zh) * 2024-02-21 2024-05-03 暗物智能科技(广州)有限公司 一种多模态信息处理方法及装置
CN117828065A (zh) * 2024-03-06 2024-04-05 深圳荣灿大数据技术有限公司 一种数字人客服方法、系统、装置及储存介质

Also Published As

Publication number Publication date
CN119156648A (zh) 2024-12-17

Similar Documents

Publication Publication Date Title
KR102364400B1 (ko) 다수의 코퍼스들로부터 응답 정보 획득
JP6745384B2 (ja) 情報をプッシュするための方法及び装置
US10771406B2 (en) Providing and leveraging implicit signals reflecting user-to-BOT interaction
CN106874467B (zh) 用于提供搜索结果的方法和装置
US20190347068A1 (en) Personal history recall
WO2024240105A1 (zh) 用于信息处理的方法、装置、设备和存储介质
WO2022254432A1 (en) A sign language translation method and system thereof
US20250148192A1 (en) Generating alternative examples for content
US20250068258A1 (en) Conversational User Interfaces With Multimodal Inputs and Mind Map Like Interactions
US12524447B2 (en) Intelligent assistant system for conversational job search
US20180137587A1 (en) Contextual personalized list of recommended courses
US20250307280A1 (en) Method, apparatus, device and storage medium for querying
WO2025241638A1 (zh) 多媒体内容推荐的方法、装置、设备和存储介质
WO2026040472A1 (zh) 应用配置方法、信息处理方法、装置、设备和存储介质
WO2026000958A1 (zh) 用于人机对话的方法、装置、设备、介质和程序产品
EP4097587A1 (en) Application search system
WO2025245697A1 (zh) 利用数字助手处理图像的方法、装置、设备和存储介质
CN119106216A (zh) 信息交互方法、装置、电子设备和存储介质
CN118828123A (zh) 显示设备和界面显示方法
US11887155B2 (en) Method and a system for selecting a targeted message to be included within a web resource
US20260030051A1 (en) System for transferring workflows between software applications
US20260105090A1 (en) Conversational computing device assistant
CN119173867A (zh) 用于查询的方法、装置、设备和存储介质
WO2026000260A1 (zh) 用于呈现多媒体内容的方法、装置、设备和介质
WO2025108115A1 (zh) 数据处理的方法、装置、设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24941015

Country of ref document: EP

Kind code of ref document: A1