CN114444003A

CN114444003A - Webpage content processing method, electronic equipment and storage medium

Info

Publication number: CN114444003A
Application number: CN202210117934.8A
Authority: CN
Inventors: 范圣强; 何贞铝; 卓奕涛; 林淦雄; 曾兴云
Original assignee: Tencent Music Entertainment Technology Shenzhen Co Ltd
Current assignee: Tencent Music Entertainment Technology Shenzhen Co Ltd
Priority date: 2022-02-08
Filing date: 2022-02-08
Publication date: 2022-05-06

Abstract

The application discloses a webpage content processing method, wherein a webpage is an application webpage running on a client, and the method comprises the following steps: initializing a scripting language component associated with a canvas element for drawing multimedia content in a webpage in response to using the canvas element when code for drawing the webpage executes; injecting a graphical application programming interface for rendering the multimedia content into the context of the scripting language component, such that the scripting language component can call the graphical application programming interface to render the multimedia content with a native client component that is a native component that runs based on an operating system of the client; and displaying the multimedia content in the application using the client component.

Description

Webpage content processing method, electronic equipment and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method for processing web page content, and a related electronic device and storage medium.

Background

With the wide development of internet technology, web pages of terminal applications are increasingly presenting richer contents, especially multimedia contents. To this end, as a user browses web pages, these web page content or resources are rendered for presentation to the user.

One conventional approach to rendering of such web page content is using a browser implementation, which requires building a multi-level tree to complete the web page layout and rendering every node of the tree.

In addition, it is also currently proposed to use the view rendering technology of ReactNative to achieve the rendering of these web page contents. The ReactNative is an open-source cross-platform mobile application development framework developed by Javascript language, and supports iOS and Android platforms. Rendering under the ReactNative framework also involves building a tree for web page layout, and rendering each node of the tree.

Accordingly, the rendering efficiency of web page content still needs to be improved, and it is desirable to reduce or avoid the rendering from unduly consuming the computing resources of the terminal and reducing the complexity of the rendering application architecture.

This background description is for the purpose of facilitating understanding of relevant art in the field and is not to be construed as an admission of the prior art.

Disclosure of Invention

Therefore, embodiments of the present invention are directed to providing a method and an apparatus for processing web content, an electronic device, and a storage medium, which can significantly improve rendering or rendering efficiency of web content with a simpler architecture.

In a first aspect, a method for processing web page content is provided, where the web page is a web page of an application running on a client, and the method includes: initializing a scripting language component associated with a canvas element for drawing multimedia content in a webpage in response to using the canvas element when code for drawing the webpage executes; injecting a graphical application programming interface for rendering the multimedia content into the context of the scripting language component, such that the scripting language component can call the graphical application programming interface to render the multimedia content with a native client component that is a native component that runs based on an operating system of the client; and displaying the multimedia content in the application using the client component.

In some embodiments, injecting the graphical application program interface into the context of the scripting language component comprises:

generating the client component; and

injecting the graphical application program interface into the context of the scripting language component based on the operating system.

In a further embodiment, generating the client component comprises:

generating a view component of the client component for processing at least layout and user interaction operations in an area of the web page where the multimedia content is displayed.

In a further embodiment, the view hierarchy of the web page is a single layer.

In a further embodiment, generating the client component further comprises:

generating a window component for the view component, the window component pointing to a graphics buffer of the client.

In a further embodiment, injecting the graphical application program interface into the context of the scripting language component based on the operating system comprises:

associating the window component with the graphical application program interface using a library file, wherein the library file is used to operate the window component, such that the graphical application program interface runs in the client component;

notifying the scripting language component that the client component is available;

causing the scripting language component to obtain a canvas instance corresponding to the window component, the canvas instance for associating the scripting language component with the graphical application program interface; and

injecting, with the canvas instance, the graphical application program interface into a context of the scripting language component based on the operating system, such that the scripting language component is capable of invoking the graphical application program interface.

In a further embodiment, injecting the graphical application program interface into the scripting language component based on the operating system using the canvas instance comprises:

determining a type of the operating system;

determining a manner of injecting the graphical application program interface into the context of the scripting language component based on the type of the operating system; and

injecting the graphical application program interface into the context of the scripting language component in the manner described using the canvas instance.

In a further embodiment, the graphical application program interface is across different operating system platforms.

In some embodiments, displaying the multimedia content with the client component comprises:

rendering the multimedia content using a graphical application program interface running in the client component;

storing the multimedia content in a graphics buffer of the client; and

causing a display device associated with the client to display the multimedia content stored in the graphics buffer.

In a second aspect, there is provided a web page content processing apparatus, the web page being a web page of an application running on a client, the apparatus comprising: an initialization module configured to initialize a scripting language component associated with a canvas element for drawing multimedia content in a webpage in response to the use of the canvas element when code for drawing the webpage executes; an interface injection module configured to inject a graphical application programming interface for rendering the multimedia content into the context of the scripting language component, such that the scripting language component can call the graphical application programming interface to render the multimedia content with a native client component, the client component being a native component that runs based on an operating system of the client; and a display module configured to display the multimedia content in the application with the client component.

In a third aspect, there is provided an electronic device comprising: a processor and a memory storing a computer program, the processor being configured to perform the processing method of any of the embodiments when the computer program is run.

In a fourth aspect, a storage medium is provided, the storage medium storing a computer program configured to perform the processing method of any of the embodiments when executed.

Embodiments of the present invention provide an improved processing scheme that enables initialization of a scripting language component associated with a canvas element in response to use of the canvas element in code for rendering a webpage of an application running on a client. The canvas element is used to draw multimedia content in the webpage. Injecting a graphical application program interface for rendering the multimedia content into the context of the scripting language component, such that the scripting language component can call the graphical application program interface to render the multimedia content with a native client component. The client component is a native component that runs based on the operating system of the client. Thereby, multimedia content can be displayed in the application using the client component. Therefore, compared with the step-by-step rendering of the view on the multi-layer tree nodes, the scheme of the embodiment of the invention realizes the high-performance webpage rendering by using a simple rendering framework.

Optional features and other effects of embodiments of the invention are set forth in part in the description which follows and in part will become apparent from the description.

Drawings

Embodiments of the invention will be described in detail, with reference to the accompanying drawings, illustrated elements not limited to the scale shown in the drawings, in which like or similar reference numerals refer to like or similar elements, and in which:

FIG. 1 shows a schematic diagram of a DOM tree of a schema for processing web page content and a view hierarchy of a web page;

FIG. 2A illustrates a schematic diagram of a web content processing environment according to an embodiment of the invention;

FIG. 2B is a diagram illustrating an exemplary architecture of a web content processing method in which embodiments of the invention may be implemented;

FIG. 3 illustrates an exemplary flow diagram of a web page content processing method according to an embodiment of the invention;

FIG. 4 illustrates an exemplary flow diagram of a method for associating a scripting language component with a client component based on an operating system, according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram illustrating a web content processing apparatus according to an embodiment of the present invention;

FIG. 6 illustrates an exemplary hardware architecture diagram of a mobile terminal capable of implementing a method according to an embodiment of the present invention;

FIG. 7 illustrates an exemplary operating system architecture of a mobile terminal capable of implementing a method in accordance with embodiments of the present invention; and

fig. 8 illustrates an exemplary operating system architecture diagram of a mobile terminal capable of implementing a method according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following detailed description and accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.

The term "include" and variations thereof as used herein is meant to be inclusive in an open-ended manner, i.e., "including but not limited to". Unless specifically stated otherwise, the term "or" means "and/or". The term "based on" means "based at least in part on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first," "second," and the like may refer to different or the same object. Other explicit and implicit definitions are also possible below.

In embodiments of the present invention, "Render" and "drawing (Paint)" have the conventional meaning of the art, especially computer graphics technology, and are used interchangeably herein to some extent, and these two terms may be used selectively according to the usual nomenclature in the art, such as "drawing an image" or "rendering a video".

The embodiment of the invention provides a webpage content processing method and device, and related electronic equipment and storage media. The web content processing method may be implemented by means of one or more computers, such as terminals, like mobile terminals, e.g. smartphones. In some embodiments, the web content processing means may be implemented by software, hardware or a combination of software and hardware.

Traditionally, the manner in which the rendering/rendering process is performed on the display content in the web page includes a default implementation of the browser. In the process of rendering the webpage by the browser, the browser receives and analyzes various requested webpage contents or resources, such as multimedia contents, and accordingly constructs a tree, performs webpage view layout, and renders each node of the tree for presenting the webpage to a user.

Another way to process the display content in the web page further includes stacking with the application development framework driving the client native components to complete the multimedia content rendering. These frameworks can be developed using scripting languages, such as JavaScript, and the multimedia content rendering interface is injected by the client into the context of the scripting language in advance for its use. When the interfaces are called, the client side is driven to create the native components, corresponding component attributes are set, and the components are combined and stacked according to the configuration of the scripting language, so that the webpage is drawn.

For example, fig. 1 shows a schematic diagram 100 of a DOM tree and a view hierarchy of a web page that processes web page content based on an open source cross-platform mobile application development framework ReactNative (ReactNative framework). To the left of FIG. 1 is a Document Object Model (DOM) tree of a scripting language view generated by the ReactNative framework, the level of the DOM tree corresponding to the right display view level 140 of a web page generated while running on the client. And each node on the DOM tree has a corresponding client native component for bearing, so that the layout and drawing of the webpage are completed. For example, as shown in FIG. 1, the DOM tree has document node 110,

view nodes

120 and 122, and text node 130, image node 132, and text node 134, and the nesting relationship of these nodes is shown as view level 140.

However, both of the above schemes have the following drawbacks.

First, web page rendering performance is poor. This is mainly reflected in the high memory space occupation of the conventional UI framework. Because each node in the tree generates a corresponding client native component for presentation, creating and holding a large number of client native components consumes more memory space, and also reduces the scroll frame rate of the entire web page.

Second, the views generated by these schemes are deeply nested, resulting in increased difficulty in developing the debug.

Again, these approaches tend to create inconsistent presentation of web page styles generated by different operating system platforms (such as iOS and Android). This is because the client components that are provided to the Web front end by different platforms are not cross-platform, requiring the client components to be packaged and injected per platform interface specifications. However, the encapsulation and injection process is manual, prone to deviation, and has poor homology. For example, in the ReactNative framework, the interaction between the native client and the JavaScript is performed through Bridge, so that the native client component and the like are packaged into a JavaScript interface through Bridge and then injected into a JavaScript engine for JavaScript calling. And ReactNative uses iOS Bridge and Android Bridge for iOS platform and Android platform respectively.

Finally, since the provision of new client components depends on the separate development of different platforms, when new client components need to be added, the encapsulation of the client components needs to be done in advance and only after sufficient testing can they be provided to the Web front end for their use. In this case, the development is less time-efficient, while it will also depend on the release of the client version.

To address, at least in part, one or more of the above problems and other potential problems, example embodiments of the present disclosure propose a solution for web page content processing. In this approach, a scripting language component, such as a JavaScript component, associated with a Canvas element is initialized in response to using the Canvas (Canvas) element in code for drawing a webpage of an application running on a client. The canvas element is used to draw multimedia content in the webpage. Injecting a graphical application program interface for rendering the multimedia content into the context of the scripting language component, such that the scripting language component can call the graphical application program interface to render the multimedia content with a native client component. The client component is a native component that runs based on the operating system of the client. Whereby the multimedia content is displayed in the application using the client component.

In an embodiment of the present invention, the Canvas (Canvas) element is an HTML tag developed using a scripting language such as JavaScript for rendering multimedia content such as images and videos. Taking an application running on a client, such as a mobile phone, as an example, the canvas element may be used to draw multimedia content in a webpage of the application. For example, in a singing application, video or audio may be rendered at a web page of the singing application to enable a user to listen to or interact with a song. In addition, images may be rendered in a web page to enable a user to share images that the user wishes to share, such as the user's business card. Currently, various browsers such as Safari, Chrome, Internet Explorer, Opera, etc. are well compatible with canvas elements. Meanwhile, the canvas element is also an interface for drawing multimedia content, which is widely used in current Web development. Therefore, the processing scheme of the embodiment of the invention performs drawing/rendering processing on the webpage elements based on the Canvas element which is a widely used element on the HTML, so that the scheme has higher universality, the Web front-end development is not limited by a specific technology, and the debugging efficiency of the development is improved.

In this manner, by associating scripting language components with native client components, a canvas interface that conforms to the Web specification (WebGL) is injected into the scripting language component context for interface calls by the Web front end. Thus, rendering multimedia content by, for example, a client component, enables the underlying use of rendering capabilities of the native client component, thereby enabling higher performance web page rendering. This stems from the fact that the inventors have realized that native client component performance is superior to browser performance, innovatively transferring the rendering tasks of a web page from the browser to the native client component.

Meanwhile, the script language component is associated with the client component based on the operating system, so that different operating system platforms are guaranteed to have completely consistent webpage performance, adaptation is not needed according to the platforms, one-time development is achieved, and cross-platform use is achieved. Therefore, the scheme solves the problem of inconsistent webpage style expressions of different operating system platforms, and obviously improves the development efficiency.

Furthermore, the scheme of the invention avoids building a multi-level tree (such as a DOM tree) and rendering each node of the tree, greatly optimizes the complexity and the real-time performance of the rendering framework, and reduces the consumption of computing resources and memory resources, thereby further improving the webpage drawing/rendering performance.

FIG. 2 shows a schematic diagram of a web content processing environment 200 according to an embodiment of the invention. It should be understood that the structure and functionality of web content processing environment 200 as shown in FIG. 2 is for illustrative purposes only and does not imply any limitation as to the scope of the present disclosure. Embodiments of the present disclosure may be embodied in different structures and/or functions.

As shown in FIG. 2A, the web content processing environment 200 may include a client 210. The client 210 may be any suitable electronic device, such as, but not limited to, a mobile phone, a tablet, a laptop, a desktop computer, a server, a mainframe, a wearable device, and so forth. A document 210 having a canvas element may be input to the client 210. As described above, canvas elements are HTML tags developed using a scripting language such as JavaScript for rendering multimedia content such as images and video. For example, the canvas element may be an HTML tag of "< canvas >". Thus, the client 210 can draw the multimedia content 230 based on the canvas element 220 and display the multimedia content 230 on a display device associated with the client 210.

Embodiments of the present disclosure will be further described below in conjunction with a web content processing environment 200 as shown in FIG. 2A. As described above, in order to improve webpage rendering performance and development efficiency and solve the problem of inconsistent webpage style expressions of different operating system platforms, an embodiment of the present disclosure proposes that an interface conforming to a canvas specification is injected into a script language component context based on an operating system, so as to be called by a Web front end, so that webpage rendering is completed by a native client component.

FIG. 3 illustrates an exemplary flow diagram of a web page content processing method 300 according to an embodiment of the invention. The actions involved in method 300 are described below in connection with web content processing environment 200 as described in FIG. 2A. Moreover, method 300 may also be implemented in the particular example architecture shown in fig. 2B, and may optionally include additional acts not shown and/or may omit acts shown, as the scope of the present disclosure is not limited in this respect. The web page is a web page of an application running on a client.

As shown in fig. 3, a web content processing method 300 according to an embodiment of the present invention may include steps 310 to 330. In an embodiment of the present invention, the web content processing method 300 may be implemented by the client 210.

Step 310: in response to using a canvas element when code for drawing a webpage executes, a scripting language component associated with the canvas element is initialized.

As described above, the canvas element is a multimedia content used to draw in a webpage of an application running on the client 210. For example, in a singing application, when a user enters a video or audio playback webpage by clicking a button, a Canvas (Canvas) element may be used to render the video or audio on the webpage to enable the user to listen to or interact with a song. Similarly, when a user wishes to generate an image for sharing, such as the user's business card, he clicks on the generate or share icon, and may draw the image in a webpage using a Canvas element to enable the user to share the image he wishes to share.

For example, the client 210, when running the canvas element 220, may initialize a scripting language component, such as a JavaScript component, associated with the canvas element.

In some embodiments, step 310 may further include loading the scripting language component, such as a JavaScript component, in a memory buffer of the client.

Step 320: injecting a graphical application program interface for rendering the multimedia content into the context of the scripting language component, such that the scripting language component can call the graphical application program interface to render the multimedia content with a native client component.

Wherein the client component is a native component that runs based on the operating system of the client 210, which is preferably used to render multimedia content. In some embodiments of the invention, the client component may be referred to as a Canvas client component. For example, the client component may be an iOS native component or an Android native component based on the client being an iOS system terminal or an Android system terminal.

In some embodiments, step 320 may include the following two steps: a1: generating a client component; a2: injecting the graphical application program interface into the context of the scripting language component based on the operating system.

In a further specific embodiment, steps a1 and a2 may each include respective sub-steps.

For example, referring to fig. 2A and 2B in combination, fig. 4 shows an exemplary flowchart of step 400 of associating a scripting language component with a client component based on an operating system, according to an embodiment of the present invention, step 400 may specifically include steps 410 through 460.

In some embodiments, the client 210 may generate a client component and inject the graphical application program interface into the context of the scripting language component based on the operating system. For example, when the scripting language component is used by the scripting language side, the client side may be notified through the UIManager interface to generate an instance of the client component. In the example shown in FIG. 2B, a Canvas client component may be generated using the Create node Command. In some embodiments, the create node command may be implemented using the HTML DOM CreateNode () method.

Step 410: a view component of the client component is generated.

In particular, the client 210 may generate a view component of the client component. In one Android-based example, as shown in FIG. 2B, the view component may be inherited to a TextureView. In one iOS-based example, the view component may be inherited to texture (asyncdisplaykit).

The view component may be used at least to handle layout and user interaction operations in areas of the web page where multimedia content is displayed, as well as other business related logic. For example, the view component may implement a timer to perform web page rendering/rendering once every predetermined time interval (e.g., 16 ms).

Step 420: a window component is generated for the view component.

In addition, client 210 may generate a window component for the view component, such as the surface component shown in FIG. 2B. The window component points to the graphics buffer of client 210. For example, the view component may obtain an instance of a window class, and the instance of the window class points to a graphics buffer, as shown in FIG. 2B.

Step 430: the window component is associated with a graphical application program interface for rendering the multimedia content using the library file such that the graphical application program interface runs in the client component.

The client 210 can associate the window component with a graphical Application Program Interface (API) for rendering the multimedia content using the library file such that the graphical application program interface runs in the client component. In this case, the client component is a container running the graphical application program interface or an environment providing the graphical application program interface to run.

In some embodiments, the graphics application program interface may be a high performance graphics application program interface across different operating system platforms, such as OpenGL. In an embodiment of the present invention, OpenGL (Open Graphics Library) is a cross-language, cross-platform Application Programming Interface (API) for rendering 2D, 3D vector Graphics. OpenGL includes OpenGL ES, which is used as a three-dimensional graphics API. In some embodiments, library files may be used to operate window components, such as an EGL library or EGL for short. EGL is an intermediate interface layer between OpenGL ES rendering APIs and native window systems (native window systems), which enables OpenGL communication with the native window systems of the device.

FIG. 2B shows that the Surface component is related to OpenGL through EGL.

Alternatively, the graphical application program interface may be other graphical application program interfaces, such as Metal, Vulkan, etc. based on the iOS platform. Some graphical application program interfaces, such as metals, that may be used in embodiments of the present invention, while defining an operating system platform, have the advantages of simple rendering architecture and high performance web page rendering as described elsewhere herein, and it is also possible to leverage the better rendering/rendering performance of these graphical application program interfaces, such as metals.

In the embodiment of the invention, the interface injected by the scheme can conform to the Web specification, so that the interface can be easily switched between any appropriate graphic application program interfaces conforming to the interface specification, such as the interfaces.

Step 440: notifying the scripting language component that the client component is available.

The client 210 may notify the scripting language component that the client component is available. In some embodiments, the client side bottom layer may notify the scripting language side layer by layer upward that a client component may be used. As shown in FIG. 2B, the scripting language component may be notified layer-by-layer upward from the beginning of the window component that the client component is available.

Step 450: the scripting language component is caused to obtain a canvas instance corresponding to the window component.

After the scripting language side receives the notification, the client 210 can cause the scripting language component to obtain the canvas instance corresponding to the window component. The canvas instance is used to associate the scripting language component with a graphics application program interface. For example, the scripting language side may use a predetermined interface (such as a getContext interface) to obtain a canvas instance corresponding to a window component. In some embodiments of the invention, as shown in FIG. 2B, this instance may also be referred to as a bound canvas (Bindingcanvas) instance, which associates a Javascript component with a client component.

S460: the graphical application program interface is injected into the context of the scripting language component based on the operating system using the canvas instance, enabling the scripting language component to invoke the graphical application program interface.

The client 210 can utilize the canvas instance to inject a graphical application program interface into the context of the scripting language component based on the operating system, enabling the scripting language component to invoke the graphical application program interface. Thus, the graphical application program interface injected into the scripting language component context can be called by the Web front end. In some embodiments, the injection operation described above is accomplished by a canvas instance, such as a Bindingcanvas.

In some embodiments, differences in different operating system platforms may be taken into account when injecting the graphics application program interface, and to this end, the client 210 may determine the type of operating system, e.g., determine whether the operating system is iOS or Android. The client 210 may determine the manner in which to inject the graphical application program interface into the context of the scripting language component based on the type of operating system, e.g., may be injected through the JavaScriptCore framework for iOS and the V8 framework for Android. Thus, the graphical application program interface can be injected into the context of the scripting language component in a determined manner. To this end, techniques for pre-compiling macros may be utilized, for example, to enable injection of graphical application program interfaces for different operating system platforms prior to compilation.

In this way, the view hierarchy of the web page is single-layered, reducing the view hierarchy complexity of the web page. Web development no longer needs to use a limited technical scheme, but can use an interface standard widely used on HTML5, namely a canvas element, to complete webpage drawing, only one layer of view hierarchy is left, and a corresponding graphics application program interface (such as OpenGL) is called for rendering, so that the development and debugging efficiency is remarkably improved. Furthermore, it is also possible to increase the homology of different operating system platforms, e.g. because the underlying layer may use a graphical application program interface (such as OpenGL) based across different operating system platforms, so the web page rendering on clients of different operating systems, such as employing iOS or Android, may be consistent.

Step 330: the client 210 displays the multimedia content 230 using a client component.

In some embodiments, the client 210 may render/render the multimedia content 230 using a graphics application program interface running in the client component, and store the rendered/rendered multimedia content 230 into a graphics buffer of the client 210 for display on the client's display screen. For example, the scripting language side may, upon invoking the canvas instance, invoke a graphics application program interface through the canvas instance according to the recorded commands to draw the multimedia content 230 and store the drawn multimedia content 230 into a graphics buffer. As such, the client 210 may cause a display device associated with the client 210 (e.g., a display device of the client 210 or a display device to which the client 210 is connected in any suitable manner) to display the multimedia content 230 stored in the graphics buffer.

The web content processing method according to the embodiment of the present disclosure is described above with reference to fig. 3 and 4. Notably, since many variables may be shared between operations associated with canvas elements and other scripting language tasks, and these variables may not be locked for thread synchronization operations, for example, in step A1, as exemplarily shown in FIG. 2B. Thus, in some embodiments, operations associated with canvas elements may be distributed to existing scripting language threads for execution, packaged into task blocks and added to a first-in-first-out task queue of the scripting language threads, and the task blocks are responsible for executing in order by the scripting language threads, thereby avoiding the problems of crashing and incorrect web page rendering caused by out-of-thread synchronization.

In this way, the present solution may also have the following benefits over a default implementation of a browser and stacking of certain frame-driven client native components to complete multimedia content rendering. For example, better web page rendering performance may be achieved. This is because, for example, the scheme can significantly reduce the memory space occupation, and at the same time, the web page rendering is transferred from the Central Processing Unit (CPU) to the Graphics Processing Unit (GPU), so that the CPU has more idle resources to complete other tasks, thereby also significantly increasing the web page frame rate. In addition, as described above, the scheme can also reduce the complexity of the view level of the webpage and increase the homology of different operating system platforms.

In an exemplary embodiment as shown in fig. 5, a web content processing apparatus 500 is also provided. The web page content processing apparatus 500 may include an initialization module 510 configured to initialize a scripting language component associated with a canvas element for drawing multimedia content in a web page of an application running on a client, in response to using the canvas element in code for drawing the web page; an interface injection module 520 configured to inject a graphical application program interface for rendering the multimedia content into the context of the scripting language component, such that the scripting language component can call the graphical application program interface to render the multimedia content with a native client component, the client component being a native component that runs based on an operating system of the client; and a first display module 530 configured to display the multimedia content in the application using the client component.

In some embodiments, the initialization module 510 may include: a generation module configured to generate a client component; and a first correlation module configured to inject the graphical application program interface into a context of the scripting language component based on the operating system.

In some embodiments, the generating module may include: a view component generation module configured to generate a view component of the client component, the view component for at least handling layout and user interaction operations in an area of the webpage displaying the multimedia content.

In some embodiments, the generating module may further include: a window component generation module configured to generate a window component for the view component, the window component pointing to a graphics buffer of the client.

In some embodiments, the first association module may include: a second association module configured to associate the window component with the graphical application program interface using a library file, such that the graphical application program interface runs in the client component, wherein the library file is used to operate the window component; a notification module configured to notify a scripting language component that a client component is available; an obtaining module configured to cause the scripting language component to obtain a canvas instance corresponding to the window component, the canvas instance for associating the scripting language component with the graphical application program interface; and a first injection module configured to inject a graphical application program interface into a context of the scripting language component based on the operating system using the canvas instance, such that the scripting language component is capable of invoking the graphical application program interface.

In some embodiments, the first injection module may include: an operating system determination module configured to determine a type of an operating system; an injection manner determination module configured to determine a manner of injecting the graphical application program interface into the context of the scripting language component based on the type of the operating system; and a second injection module configured to utilize the canvas instance to inject a graphical application program interface in the context of a scripting language component in a manner.

In some embodiments, the graphical application program interface is across different operating system platforms.

In some embodiments, the first display module 530 includes: a rendering module configured to render the multimedia content using a graphical application program interface running in the client component; a storage module configured to store multimedia content into a graphics buffer of a client; and a second display module configured to cause a display device associated with the client to display the multimedia content stored in the graphics buffer.

In an embodiment of the present invention, there is provided an electronic apparatus including: a processor and a memory storing a computer program, the processor being configured to implement any of the methods according to embodiments of the invention when running the computer program. In addition, a processing apparatus implementing the embodiment of the present invention may also be provided.

In a preferred embodiment of the present invention, the electronic device is a mobile terminal, which may preferably be a mobile phone. By way of exemplary implementation only, FIG. 6 illustrates a hardware architecture diagram of a particular embodiment of an electronic device, such as a mobile terminal 600; and figures 7 and 8 show system architecture diagrams of a particular embodiment of an electronic device, such as a mobile terminal. The terminal examples shown in fig. 6 to fig. 8 may also be used to implement the client according to the solution of the embodiment of the present invention.

In the illustrated embodiment, the mobile terminal 600 may include a processor 601, an external memory interface 612, an internal memory 610, a Universal Serial Bus (USB) interface 613, a charge management module 614, a power management module 615, a battery 616, a mobile communication module 640, a wireless communication module 642, antennas 639 and 641, an audio module 634, a speaker 635, a microphone 636, a microphone 637, an earphone interface 638, keys 609, a motor 608, an indicator 607, a Subscriber Identity Module (SIM) card interface 611, a display 605, a camera 606, a sensor module 620, and so forth.

It is to be understood that the illustrated structure of the embodiment of the present application does not constitute a specific limitation to the mobile terminal 600. In other embodiments of the present application, the mobile terminal 600 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

In some embodiments, processor 601 may include one or more processing units. In some embodiments, the processor 601 may include one of the following or a combination of at least two of the following: an Application Processor (AP), a modem processor, a baseband processor, a Graphics Processor (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, a neural Network Processor (NPU), and so forth. The different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural center and a command center of the mobile terminal 600. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.

A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor. If the processor needs to reuse the instruction or data, it can be called directly from memory. Avoiding repeated accesses reduces the latency of the processor 601 and thus increases the efficiency of the system.

The NPU is a Neural Network (NN) computational processor that processes input information quickly by referencing a biological neural network structure, such as by referencing transfer patterns between human brain neurons, and may also be continuously self-learning.

The GPU is a microprocessor for image processing and is connected with a display screen and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs that execute program instructions to generate or alter display information.

The digital signal processor (ISP) is used to process digital signals and may process other digital signals in addition to digital image signals.

In some embodiments, the processor 601 may include one or more interfaces. The interfaces may include an integrated circuit (I2C) interface, an integrated circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General Purpose Input Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, a Universal Serial Bus (USB) interface, and so forth.

It should be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only an exemplary illustration, and does not constitute a limitation to the structure of the mobile terminal. In other embodiments of the present application, the mobile terminal may also adopt different interface connection manners or a combination of multiple interface connection manners in the foregoing embodiments.

The wireless communication function of the mobile terminal 600 may be implemented by the antennas 639 and 641, the mobile communication module 640, the wireless communication module 642, a modem processor, a baseband processor, or the like.

Video codecs are used to compress or decompress digital video.

The mobile terminal 600 may implement audio functions through an audio module, a speaker, a receiver, a microphone, an earphone interface, an application processor, and the like. Such as music playing, recording, etc.

The audio module is used for converting digital audio information into analog audio signals to be output and converting analog audio input into digital audio signals.

The microphone is used for converting a sound signal into an electric signal. When making a call or sending voice information, a user can input a voice signal into the microphone by making a sound by approaching the microphone through the mouth of the user.

The sensor module 620 may include one or more of the following sensors:

pressure sensor 623 is configured to sense a pressure signal and convert the pressure signal into an electrical signal.

The air pressure sensor 624 is used to measure air pressure.

The magnetic sensor 625 includes a hall sensor.

The gyro sensor 627 may be used to determine the motion pose of the mobile terminal 600.

The acceleration sensor 628 may detect the magnitude of acceleration of the mobile terminal 600 in various directions.

The distance sensor 629 may be configured to measure distance.

The sensor module 620 may further include a gravity sensor 626 and a temperature sensor 630.

The proximity light sensor 621 may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode.

The ambient light sensor 622 is used to sense ambient light levels.

The fingerprint sensor 631 may be configured to collect a fingerprint.

The touch sensor 632 may be disposed on the display screen, and the touch sensor and the display screen form a touch screen, which is also called a "touch screen". The touch sensor is used to detect a touch operation applied thereto or nearby. The touch sensor can communicate the detected touch operation to the application processor to determine a touch event type, such as a single click, a double click, a long press, a tap, a directional swipe, a bunch, and so forth.

Bone conduction sensor 633 may acquire a vibration signal.

A software operating system of an electronic device (computer), such as a mobile terminal, may employ a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture.

The embodiments illustrated herein exemplify the software structure of a mobile terminal, taking the iOS and android operating system platforms, respectively, as a layered architecture. It is contemplated that embodiments herein may be implemented in different software operating systems.

In the embodiment shown in fig. 7, the solution of the embodiment of the present invention may employ an iOS operating system. The iOS operating system adopts a four-layer architecture, which comprises a touchable layer (coco Touch layer)710, a Media layer (Media layer)720, a Core Services layer (Core Services layer)730 and a Core operating system layer (Core OS layer)740 from top to bottom. The touch layer 710 provides various common frameworks for application development and most of the frameworks are related to interfaces, which are responsible for user touch interaction operations on the iOS device. The media layer provides the technology of audio-visual aspects in the application, such as graphic images, sound technology, video and audio-video transmission related frameworks and the like. The core service layer provides the underlying system services required by the application. The core operating system layer contains most of the low level hardware-like functionality.

In an embodiment of the present invention, UIKit is a user interface framework of the touchable layer 710 that can be supported by numerous Image frames in the media layer 720, including but not limited to the Core gallery (Core Graphics), Core Animation (Core Animation), open gallery es (open GL es), Core map (Core Image), Image io (imageio), gallery package (GLKit) shown in fig. 7.

Fig. 8 is a schematic structural diagram of an android operating system, which may be adopted in the solution of the embodiment of the present invention. The layered architecture divides the software into several layers, which communicate via software interfaces. In some embodiments, the android system is divided into four layers, from top to bottom, an application layer 810, an application framework layer 820, an android Runtime (Runtime) and system library 830, and a kernel layer 840.

The application layer 810 may include a series of application packages.

The application framework layer 820 provides an Application Programming Interface (API) and a programming framework for applications of the application layer. The application framework layer includes a number of predefined functions.

The window manager is used for managing window programs.

The content provider is used to store and retrieve data and make it accessible to applications.

The view system includes visual controls such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide a communication function of the mobile terminal.

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

The notification manager enables the application to display notification information in the status bar, can be used to convey notification-type messages, can disappear automatically after a short dwell, and does not require user interaction.

The android Runtime comprises a core library and a virtual machine, and is responsible for scheduling and managing an android system. The core library comprises two parts: one part is a function to be called by java language, and the other part is a core library of android. The application layer and the framework layer run in a virtual machine.

The system library may include a plurality of functional modules. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

The media library supports a variety of commonly used audio, video format playback and recording, and still image files, among others. The media library may support a variety of audio-video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The kernel layer 840 is a layer between hardware and software. The kernel layer may include a display driver, a camera driver, an audio interface, a sensor driver, power management, and a GPS interface. In some embodiments of the invention, the display may invoke display driving.

The systems, apparatuses, modules or units illustrated in the above embodiments may be implemented by an electronic device (computer) or its associated components, preferably a mobile terminal. The mobile terminal may be, for example, a smart phone, a laptop computer, a vehicle human interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet, a wearable device, or a combination thereof.

Although not shown, in some embodiments a storage medium is also provided, storing the computer program. The computer program is configured to perform the method of any of the embodiments of the invention when executed.

Storage media in embodiments of the invention include permanent and non-permanent, removable and non-removable articles of manufacture in which information storage may be accomplished by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Thus, it will be apparent to one skilled in the art that the implementation of the functional modules/units or controllers and the associated method steps set forth in the above embodiments may be implemented in software, hardware, and a combination of software and hardware.

Unless specifically stated otherwise, the actions or steps of a method, program or process described in accordance with an embodiment of the present invention need not be performed in a particular order and still achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

While various embodiments of the invention have been described herein, the description of the various embodiments is not intended to be exhaustive or to limit the invention to the precise forms disclosed, and features and components that are the same or similar to one another may be omitted for clarity and conciseness. As used herein, "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" are intended to apply to at least one embodiment or example, but not to all embodiments, in accordance with the present invention. The above terms are not necessarily meant to refer to the same embodiment or example. Various embodiments or examples and features of various embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Exemplary systems and methods of the present invention have been particularly shown and described with reference to the foregoing embodiments, which are merely illustrative of the best modes for carrying out the systems and methods. It will be appreciated by those skilled in the art that various changes in the embodiments of the systems and methods described herein may be made in practicing the systems and/or methods without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for processing web page content, wherein the web page is a web page of an application running on a client, the method comprising:

initializing a scripting language component associated with a canvas element for drawing multimedia content in a webpage in response to using the canvas element when code for drawing the webpage executes;

injecting a graphical application programming interface for rendering the multimedia content into the context of the scripting language component, such that the scripting language component can call the graphical application programming interface to render the multimedia content with a native client component that is a native component that runs based on an operating system of the client; and

displaying the multimedia content in the application with the client component.

2. The method of claim 1, wherein injecting the graphical application program interface into the context of the scripting language component comprises:

generating the client component; and

3. The method of claim 2, wherein generating the client component comprises:

4. The method of claim 3, wherein the view hierarchy of the web page is a single layer.

5. The method of claim 3 or 4, wherein generating the client component further comprises:

6. The method of claim 5, wherein injecting the graphical application program interface into the context of the scripting language component based on the operating system comprises:

associating the window component with the graphical application program interface using a library file, such that the graphical application program interface runs in the client component, wherein the library file is used to manipulate the window component;

7. The method of claim 6, wherein injecting, with the canvas instance, the graphical application program interface into the scripting language component based on the operating system comprises:

determining a type of the operating system;

injecting the graphical application program interface into the context of the scripting language component in the manner described with the canvas instance.

8. The method of claim 6, wherein the graphical application program interface is across different operating system platforms.

9. The method of any of claims 6 to 8, wherein displaying the multimedia content with the client component comprises:

storing the multimedia content in a graphics buffer of the client; and

10. An electronic device, comprising: a processor and a memory storing a computer program, the processor being configured to perform the method of any of claims 1 to 9 when the computer program is run.

11. A storage medium, characterized in that the storage medium stores a computer program configured to perform the method of any one of claims 1 to 9 when executed.