WO2021227628A1

WO2021227628A1 - Electronic device and interaction method therefor

Info

Publication number: WO2021227628A1
Application number: PCT/CN2021/079995
Authority: WO
Inventors: 提纯利
Original assignee: 华为技术有限公司
Priority date: 2020-05-14
Filing date: 2021-03-10
Publication date: 2021-11-18
Also published as: CN113672099A

Abstract

The present application is suitable for the technical field of handwriting interactions, and provides an interaction method for an electronic device, comprising: an electronic device displays an interaction image; the electronic device obtains a first operation of a stylus according to movement information of the stylus, wherein the movement information comprises relative movement information of the stylus; the electronic device displays a virtual interaction interface on the interaction image, and responds to the first operation on the virtual interaction interface. By obtaining the relative movement information of the stylus, detailed relative displacement detection can be performed on the stylus, so that a writing trajectory of the stylus can be accurately determined, and the writing trajectory can be accurately restored, thus facilitating realizing an accurate interaction operation.

Description

Electronic equipment and its interaction method

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on May 14, 2020, the application number is 202010407584.X, and the application name is "an electronic device and its interaction method", the entire content of which is incorporated by reference In this application.

Technical field

This application belongs to the technical field of handwriting interaction, and in particular relates to an electronic device and an interaction method thereof.

Background technique

Industries such as virtual reality, augmented reality, and mixed reality are gradually rising with the development of 5G transmission technology, display technology and interactive technology. For example, mixed reality technology generates a new visual environment by merging reality and virtual world. By introducing real scene information into the virtual environment, an interactive feedback information loop is built between the virtual world, the real world and the user to enhance the user The realism of the experience.

Emerging interactive technologies for mixed reality scenes such as data gloves can better target game and entertainment scenes, but the ability to input text content through data gloves is weak. However, the operation method of collecting the image of the stylus pen to detect the pen tip through the visual algorithm cannot accurately determine the position of the pen tip, and therefore cannot accurately restore the writing trajectory.

Summary of the invention

The embodiments of the present application provide an electronic device and an interactive method thereof, which can solve the problem of the inability to accurately determine the position of the pen tip when interactive input is performed through a stylus in the prior art.

In the first aspect, an embodiment of the present application provides an interactive method for an electronic device. The electronic device displays an interactive image. The interactive image may be an image of the current scene or an image of a multimedia file; The movement information of the pen acquires the first operation of the stylus, wherein the movement information includes the relative movement information of the stylus, and the first operation may include a tap operation, a writing operation, or may also include the tip of the stylus in the empty state Mobile operation; the electronic device displays a virtual interactive interface on the interactive image, and responds to the first operation on the virtual interactive interface. When the first operation is a click operation, a click can be triggered in the virtual interactive interface Instruction, when the first operation is a writing operation, the corresponding writing track or writing content may be displayed in the virtual interactive interface.

The electronic device is used to display an interactive image and a virtual interactive interface, can receive interactive data input by an input device such as a stylus pen, and respond to the interactive data in the displayed virtual interactive interface. The electronic device may be an electronic device such as a head-mounted display device, virtual reality glasses, augmented reality glasses, or mixed reality glasses. The relative motion information can be collected by sensing devices such as gyroscopes, inertial sensors, and acceleration sensors. By acquiring the relative movement information of the stylus, the electronic device can perform more detailed relative displacement detection of the stylus, so that the writing trajectory of the stylus can be determined more accurately, and the writing trajectory can be restored more accurately. Facilitate the realization of more refined interactive operations. In a possible implementation manner, the sensing device is set at the tip position of the stylus pen, and by collecting the relative movement information of the tip position, the writing track of the stylus pen can be obtained more accurately.

In an implementation manner, the interactive image is an image of the current scene, and the writing position of the stylus can be determined by the correspondence between the image of the current scene and the interactive image. It may include: acquiring the position of the stylus in the current scene according to the image of the current scene, wherein the position of the stylus in the current scene can be represented by the relative positional relationship of the stylus with respect to the handwriting plane; The position of the stylus in the current scene determines the handwriting position of the tip of the stylus in the virtual interactive interface. The relative position of the stylus with respect to the handwriting plane can be determined by combining with the preset mapping relationship. The corresponding position of the pen tip in the virtual interactive interface.

While determining the writing trajectory of the stylus, the position of the stylus in the current scene can also be determined by locating the position of the stylus in the current scene, and the handwriting position to be written in the empty state can be tracked. For example, the corresponding position of the stylus in the virtual interactive interface can be tracked in real time by moving virtual icons, including virtual stylus pens and virtual cursors, so as to improve the convenience of user writing.

In the implementation of determining the position of the stylus in the current scene, it may include: acquiring the image of the current scene according to the camera, and directly determining the position of the stylus in the current scene through the image of the current scene; or, also including acquiring the current scene The depth information, combined with the depth information, can obtain the position of the stylus in the current scene more accurately.

When the image of the current scene is collected by the camera, the initial position of the stylus in the image of the current scene can be recognized according to the preset characteristics of the stylus. For example, the initial position of the stylus can be determined according to the position of the tip of the stylus in the image of the current scene relative to the position of other reference information in the image. The other reference information may be information such as the edges and vertices of the writing plane, characters or patterns included in the writing plane. Or, in an implementation manner, the image of the current scene and the depth information of the current scene can be combined to determine the position of the pen tip, the position of the writing plane, and the position relative to the writing plane. According to the determined relative position, the position of the writing pen can be obtained. The corresponding position in the virtual interactive interface.

In an implementation manner for determining the position of the stylus in the current scene, it may include: detecting the tip feature of the stylus in the image of the current scene, and determining the position of the tip of the stylus in the image of the current scene; The matching relationship between the image of the current scene and the depth information to obtain the depth information matched by the pen tip; according to the depth information of the pen tip, the spatial coordinates of the pen tip can be determined, and the spatial coordinates of the determined handwriting plane can be combined to obtain the The position of the pen tip relative to the handwriting plane.

By acquiring the image of the current scene and combining the characteristics of the pen tip, the position of the pen tip in the image of the current scene can be determined. Through the image of the current scene and the depth information corresponding to the image of the current scene, the depth information of the object in the image of the current scene can be determined. Combined with the depth information of the tip of the stylus, the relative position of the tip of the stylus with respect to the writing plane can be determined. According to the relative position, the writing position of the stylus corresponding to the virtual interactive interface can be effectively determined.

In a method for setting the pen tip feature of a stylus pen, the pen tip feature includes one or more of color feature, reflected light feature, or fluorescent feature. Including the combination of color feature and reflected light feature, combination of reflected light feature and fluorescence feature, or combination of color feature and fluorescence feature, or combination of color feature, fluorescence feature and reflected light feature, etc.

By setting the pen tip of the stylus to a specific color, the position of the pen tip in the image of the current scene can be quickly obtained through color detection. The color of the pen tip may be different from the color of the writing plane. In an implementation manner, the color of the current writing plane can be detected, and the color of the pen tip can be adjusted according to the color of the current writing plane, so as to facilitate adapting to the pen tip detection requirements of different writing planes. Alternatively, the pen tip may be set as a reflective material layer, the light is reflected by the reflective material, the position of the reflected light is detected in the image of the current scene, and the position of the pen tip is determined. Alternatively, the pen tip may be provided with a fluorescent material layer, and the position of the pen tip can be determined by detecting the fluorescent position in the image of the current scene.

In an implementation manner of determining the handwriting position of the pen tip in the virtual interactive interface, it may include: when the stylus is in the empty state, acquiring the position of the stylus and the position of the writing plane of the current scene, and determining the handwriting The relative positional relationship of the pen relative to the writing plane; according to the relative positional relationship of the stylus relative to the writing plane, combined with the predetermined mapping relationship between the writing plane and the virtual interactive interface, it is determined that the stylus is in the virtual interactive interface s position.

By pre-setting the positional relationship between the writing plane and the virtual interactive interface, when the stylus is in the empty state, by acquiring the position of the stylus and the position of the writing plane, the position of the stylus relative to the writing plane can be determined. Combined with the mapping relationship between the writing plane and the virtual interactive interface, the position of the stylus in the virtual interactive interface can be determined. By detecting the position of the stylus when the stylus is in the empty state, it can be determined that the writing position of the stylus in the empty state changes in the virtual interactive interface, which is convenient for the user to observe in real time when the stylus is down, which corresponds to the virtual interface. Writing position.

In an interactive image generation method, the interactive image is collected by a camera or presented by a display device with a predetermined transparency, that is, according to the set transparency, the image of the real scene is allowed to partially penetrate the display device, and the user The image of the real scene can be viewed through the display device with the predetermined transparency.

Among them, the interactive image collected by the camera may be an image of the user's current scene. The image content of the interactive image may correspond to the vision range of the naked eye when the user is not wearing the electronic device, so that it can be mixed with the virtual interactive interface to obtain a mixed reality image. Alternatively, the interactive image may be displayed in a display device with a predetermined transparency. The predetermined transparency display device may be related to scene information. For example, the transparency of the display device may gradually decrease when the scene light intensity increases, and when the scene light intensity decreases, the transparency gradually increases, so that an interactive image with more comfortable brightness can be obtained.

In the implementation of displaying the virtual interactive interface, it may include displaying the virtual interactive interface in a predetermined area in the interactive image, that is, the display position of the virtual interactive interface is fixed, which is convenient for recording general information of the screen, such as recording the current scene Information such as the feeling, mood or date corresponding to the image; or, the position of the virtual interactive interface is determined according to the plane area in the current scene, so as to facilitate the coordination of the unification of the writing position and the display position of the writing track.

By displaying the virtual interactive interface in a fixed area in the interactive image, the interactive image can be selected more flexibly. Write in the outer area, and display the written information in the interactive image. At this time, the position information of the stylus can be collected through sensors such as a camera.

When the position of the virtual interactive interface is determined by the plane area in the current scene, the writing information can be displayed at the position of the stylus, which can make the correspondence between the writing position and the displayed content more in line with the user's usage habits, and improve the convenience of the user. sex.

When determining the position of the virtual interactive interface according to the plane area in the current scene, it may include: filtering the plane area in the current scene according to preset plane area requirements; according to the shape and/or position of the filtered plane area, Determine the shape and/or position of the virtual interactive interface. Wherein, the preset requirement may include one or more of the size range of the plane area, the orientation of the plane area, or the distance of the plane area from the camera.

For example, by pre-setting the size range of the plane area, the plane area with a smaller size can be screened, and the plane area with a larger size or meeting the requirements of the size range can be automatically selected, so that it is convenient for the user to write. According to the orientation requirements of the plane area, the plane orientation can be selected as upward or toward the plane area of the user to improve the convenience of the user's writing. By filtering the distance between the plane area and the camera, the plane area that is far away and inconvenient for the user to write can be filtered out, and a plane area that is more convenient for the user to write can be obtained.

In the implementation of screening the planes, when there are multiple plane areas that meet the requirements of the plane area, the method further includes: selecting among the plurality of plane areas according to the predetermined preferred writing position of the stylus. A plane area that better matches the better writing position.

Wherein, when selecting a plane area according to the determined preferred writing position, it may include: respectively obtaining the distances between a plurality of to-be-selected plane areas and the preferred writing position; and selecting a plane area with a relatively close distance.

Alternatively, obtain the preferred writing area corresponding to the preferred writing position; respectively obtain the intersection area of a plurality of to-be-selected plane areas and the preferred writing area; select the to-be-selected plane area with a larger area of the intersection area.

The preferred writing position can be set by the user, or it can be obtained by statistical analysis of the preferred writing position according to the user's writing habits, that is, the user's writing image is analyzed to determine the position where the user writes more frequently, and the The determined position is used as a better writing position.

When determining the distance between the writing plane and the preferred writing position, it can be the closest distance between the edge of the writing plane and the preferred writing position, or it can also be the distance between the center of the writing plane and the preferred writing position. The size of the preferred writing area can be specified by the user, or the preferred writing range can be obtained by automatic statistics according to the user's writing habits.

In an implementation manner, the method further includes: displaying an edit button in the virtual interactive interface; responding to the writing information on the virtual interactive interface includes: when a stylus is detected at a corresponding position of the edit button During the click operation, respond to the corresponding function of the edit button.

By displaying the virtual keys in the virtual interactive interface and clicking the virtual keys through the stylus, the functions corresponding to the virtual keys are realized. For example, the virtual buttons may include buttons such as a brush shape and a brush function. By clicking the button, the image drawn by the writing pen can be edited, or the information or image generated by the image of the current scene can be edited.

In the implementation of displaying virtual keys in the virtual interactive interface, it may also include saving and/or sending the edited image or text content to other users; or, selecting the text content of the virtual interactive interface and sending the search place to the network. A request for selected text content; receiving and displaying the result corresponding to the request on the virtual interactive interface.

By triggering the virtual button, the interactive image and the image of the virtual interactive interface can be saved, and the real-time saved image can be transmitted to other users through the network. Alternatively, the electronic device can also collect audio information in real time, and transmit the audio information and saved images to other users in real time, so that convenient online teaching and explanation can be realized.

By selecting the text content of the virtual interactive interface, the search request is triggered by the virtual button, and the result corresponding to the request is received and displayed, which can facilitate the user to interact with the network in real time. The text content of the virtual interactive interface may be the content of text media in the real scene automatically recognized by the electronic device, thereby facilitating more convenient query of the realization scene. Alternatively, the text content of the virtual interactive interface may also be text content included in a multimedia image.

In an implementation manner for calibrating the writing accuracy of an electronic device, the method includes obtaining an image of the pen tip when drawing is performed through a visual perception module, and generating a first trajectory according to the image when drawing is performed; acquiring through a relative motion sensing unit The relative displacement information of the pen tip when the drawing is executed, the second trajectory is generated according to the relative displacement information; the difference between the first trajectory and the second trajectory is compared, and the relative motion information is calibrated according to the difference .

The first trajectory is generated by the visual perception module, and the second trajectory is generated by the relative motion information. The relative motion information is calibrated by comparing the difference between the first trajectory and the second trajectory, for example, the relative displacement information is obtained by sensing equipment Adjustment, which can improve the accuracy of the relative movement. When the first trajectory is acquired through the visual perception module, the position of the stylus pen tip in the three-dimensional scene model can be determined by establishing a three-dimensional scene model, according to the position of the stylus in the interactive image and the depth information of the interactive image, and thereby Get a more accurate first trajectory.

In the second aspect, an embodiment of the present application provides an electronic device that includes a memory, a processor, and a computer program that is stored in the memory and can run on the processor, and the processor executes all The computer program implements the interactive method of the electronic device as described in any one of the first aspect.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the Interactive method of electronic equipment.

The electronic device described in the second aspect and the readable storage medium described in the third aspect correspond to the interaction method of the electronic device described in the first aspect.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only of the present application. For some embodiments, those of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic block diagram of the structure of an electronic device according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a use state of an electronic device provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of a display of a virtual interactive interface provided by an embodiment of the present application;

4 is a schematic diagram showing another virtual interactive interface provided by an embodiment of the present application;

Fig. 5 is a schematic diagram of determining a virtual interactive interface according to a preferred writing position according to an embodiment of the present application;

6 is another schematic diagram of determining a virtual interactive interface according to a preferred writing position according to an embodiment of the present application;

FIG. 7 is a schematic diagram of the layout of a head-mounted display device provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of a virtual interactive interface provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of the implementation process of an electronic device interaction method provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of image acquisition of a main device provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of depth information calculation provided by an embodiment of the present application;

FIG. 12 is a schematic diagram of detecting a current scene image provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of determining the position of a pen tip provided by an embodiment of the present application;

FIG. 14 is a schematic diagram of an image taken by a visual perception device provided by an embodiment of the present application;

FIG. 15 is a schematic diagram of a virtual interactive interface provided by an embodiment of the present application;

FIG. 16 is a schematic diagram of a device calibration provided by an embodiment of the present application;

FIG. 17 is a schematic diagram of a screen of an electronic device used for signing according to an embodiment of the present application;

FIG. 18 is a schematic diagram of a screen of an electronic device used for live broadcasting according to an embodiment of the present application;

FIG. 19 is a schematic diagram of a screen for writing on an electronic device according to an embodiment of the present application;

FIG. 20 is a schematic diagram of an interaction apparatus of an electronic device provided by an embodiment of the present application.

Detailed ways

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are proposed for a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obstructing the description of this application.

The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also This includes expressions such as "one or more" unless the context clearly indicates to the contrary. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and/or" describes the association relationship of the associated objects, indicating that there may be three relationships; for example, A and/or B can mean the situation where A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character "/" generally indicates that the associated objects before and after are in an "or" relationship.

The electronic device interaction method provided by the embodiments of this application can be applied to augmented reality (AR)/virtual reality (VR) devices, mobile phones, tablet computers, wearable devices, in-vehicle devices, notebook computers, and ultra mobile On devices such as ultra-mobile personal computers (UMPC), netbooks, and personal digital assistants (personal digital assistants, PDAs), the embodiments of this application do not impose any restrictions on the specific types of electronic devices.

For example, the electronic device may be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a wireless local loop (Wireless Local Loop, WLL) station, Personal Digital Assistant (PDA) devices, handheld devices with wireless communication functions, computing devices or other processing devices connected to wireless modems, in-vehicle devices, car networking terminals, computers, laptop computers, handheld communication devices , Handheld computing devices, satellite wireless devices, wireless modem cards, television set top boxes (STB), customer premise equipment (customer premise equipment, CPE), and/or other equipment used to communicate on the wireless system and download A first-generation communication system, for example, a mobile terminal in a 5G network or a mobile terminal in a public land mobile network (PLMN) network that will evolve in the future.

As an example and not a limitation, when the electronic device is a wearable device, the wearable device can also be a general term for using wearable technology to intelligently design daily wear and develop wearable devices, such as glasses, gloves, Watches, clothing and shoes, etc. A wearable device is a portable device that is directly worn on the body or integrated into the user's clothes or accessories. Wearable devices are not only a kind of hardware device, but also realize powerful functions through software support, data interaction, and cloud interaction. In a broad sense, wearable smart devices include full-featured, large-sized, complete or partial functions that can be achieved without relying on smart phones, such as smart watches, head-mounted display devices, or smart glasses, and only focus on a certain type of application function. Equipment such as smart phones are used together, such as various smart bracelets and smart jewelry for physical sign monitoring.

In an implementation manner, the electronic device includes a main device and a stylus. FIG. 1 shows a schematic block diagram of a part of the structure of an electronic device provided in an embodiment of the present application. 1, the electronic device includes a head-mounted display device 1 as a main device and a stylus 2 that can establish a connection with the main device. Wherein, the head-mounted display device 1 includes a first communication unit 110, a visual perception module 120, a depth sensing unit 130, a display unit 140, a first calculation processing unit 150, a first storage unit 160, and a first power supply unit 180 The stylus 2 includes a second communication unit 210, a relative motion sensing unit 220, a second calculation processing unit 230, a second storage unit 240, and a second power supply unit 250. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 1 does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, or a combination of certain components, or different component arrangements.

The following is a detailed introduction to the various components of the electronic device with reference to Figure 1:

The first communication unit 110 can communicate with the second communication unit 210. Alternatively, the first communication unit 110 and the second communication unit 210 may use short-distance communication circuits, including but not limited to Bluetooth communication circuits, infrared communication circuits, Wifi communication circuits, and the like. The first communication unit 110 can establish a connection link with the second communication unit 210. The first communication unit 110 or the second communication unit 210 may also establish a communication connection with other electronic devices. For example, the first communication unit 110 and the second communication unit 210 can establish a communication link with devices such as smart phones, computers, etc., and send data collected or processed by the head-mounted display device 1 or the stylus 2 to other electronic devices. The device, or through the link, the head-mounted display device 1 or the stylus 2 receives data sent by other electronic devices.

For example, in the schematic diagram of the use state of the electronic device shown in FIG. 2, the user can fix the head-mounted display device 1 on the user's head by wearing it, and the head-mounted display device 1 can use a camera The way of collecting images, and displaying the collected images of the current scene on the display screen of the head-mounted display device in real time. For example, the captured image can be divided into a first image and a second image, which are respectively displayed on the first display screen and the second display screen in the head-mounted display device, so that the user can watch through the head-mounted display device To the real image in the current scene.

Alternatively, the display screen of the head-mounted display device may be a screen with a predetermined transparency, such as a display screen with a semi-transparent structure. The user can view the picture in the current scene in real time through the light transmitted through the display screen.

At the same time, a virtual screen may also be displayed on the display screen in the head-mounted display device, and the virtual screen may include a virtual interactive interface. For example, the virtual interactive interface may display the handwriting written by the stylus held by the user shown in FIG. 2. Alternatively, the virtual interactive interface may display the text content obtained by performing text recognition on the text medium in the current scene. Alternatively, the virtual interactive interface may also include an application program interface opened by the user, and screen content such as editing and processing of written characters and images.

The head-mounted display device 1 is provided with a first communication unit 110, and the stylus 2 is provided with a second communication unit 210. The first communication unit 110 and the second communication unit 210 may establish a link connection via Bluetooth.

The stylus 1 is provided with a sensing device for detecting relative movement, and can collect relative displacement data of the stylus. The relative displacement data may include the position of the tip of the stylus relative to the last detection time, the distance and direction of the movement of the tip of the stylus at the current detection moment, and the collected relative displacement data is sent through the link Give head-mounted display devices. That is, when the user uses the stylus pen to write text, draw or other editing processing actions, if the stylus is detected to be in the writing state, the detection is performed according to the preset time interval of every two adjacent detection moments to obtain each The relative displacement data of the detection time relative to the previous detection time, that is, the relative displacement distance and the relative displacement direction of the current detection time relative to the previous detection time are obtained. According to the relative displacement data, combined with the pen tip position at the previous detection time, the pen tip position at the current detection time can be determined. After determining the position of the pen tip at each detection moment, the pen tip trajectory when the user writes with the stylus pen can be obtained.

In an implementation manner, the stylus 2 can also detect whether the stylus is in a writing state or an empty state through a pressure sensor, and send the detected stylus state information to the head-mounted display device through the link, Or, the relative displacement data of the stylus relative to the handwriting plane can also be collected by the stylus pen according to the state information of the stylus pen.

The visual perception module 120 may be a visible light camera. The camera can capture external environment information to generate a video stream signal, and provide data for the current scene mapping (Mapping), stylus localization (localization), gestures, stylus gestures and other action recognition. In an implementation manner, multiple cameras can be set to collect images in the scene at different angles of view to provide multi-view stereo vision for current scene drawing. Wherein, the current scene drawing may include drawing one or more of the objects included in the current scene, the image of the object, the size of the object, the position of the object in the current scene, the distance between the object and the user, and other information.

In an implementation manner, the main device includes a first camera 1201 and a second camera 1202, where the first camera may be a visible light camera, and the second camera may be a visible light camera or an infrared camera. The images collected by the first camera and the second camera are combined with the camera parameters of the first camera and the second camera, including the internal parameters of the camera and the external parameters of the camera, to determine the depth information of the object in the image. According to the determined depth information of the object in the image, it can be used to determine the position of the object in the image of the current scene, or to determine the distance between the object and the user when the current scene is drawn.

In an implementation manner, the head-mounted display device 1 may further include a light supplement unit 180. The light supplement unit 180 can provide visible light supplement light. When the second camera is an infrared camera, the supplementary light unit 180 may also provide infrared supplementary light. Through infrared supplement light, and the infrared camera for image collection, it can effectively improve the accuracy and robustness of the pen tip's visual positioning without affecting the perception of the environment.

The depth sensing unit 130 is used to detect the distance between an object in the current scene and the head-mounted display device 1. The depth sensing unit may include two or more cameras. Alternatively, the depth sensing unit 130 may include one or more of distance measurement units such as a time-of-flight ranging camera, a structured light ranging system, a radar, and an ultrasonic sensor.

Through the depth information of the object detected by the depth sensing unit 130, combined with the image collected by the camera, the current scene can be 3D modeled, the plane in the current scene can be detected, and the real-time location and map of the scene can be completed Build SLAM (Simultaneous Localization And Mapping), etc.

When the depth sensing unit 130 is two cameras, the depth information of the object in the image is determined according to the images captured by the two cameras and the parameter information of the two cameras.

When the depth sensing unit 130 is a single camera, it can record the emission time of the light beam by emitting a light beam to the object. In addition, when the camera captures the light beam reflected by the object, the receiving time of the recording light beam is recorded, and the distance between the object and the camera can be calculated according to the time difference between the emission time and the receiving time, combined with the propagation speed of the light.

The display unit 140 may be used to display an interactive image, and the interactive image may be a video image taken by a camera, or may also include a virtual interactive interface. Or the display unit 140 is a display device with a predetermined transparency. The predetermined transparency may be a semi-transparent display device through which the user can see the current scene, and superimpose and display a virtual interactive interface on the semi-transparent display device. The predetermined transparency can also be automatically changed according to the brightness of the current scene. For example, when the brightness of the current scene increases, the transparency of the display device can be reduced, and when the brightness of the current scene decreases, the transparency of the display device can be increased, so that the user can view interactive images with appropriate brightness through the display unit.

Fig. 3 is a schematic diagram showing the display of a virtual interactive interface provided by an embodiment of the application. In an implementation manner as shown in FIG. 3, the virtual interactive interface 302 may be superimposed and displayed on an interactive image, such as the current scene image 301. The virtual interactive interface 302 may be an area where interactive content is displayed, such as an area where the track of a stylus pen and a cursor corresponding to the stylus pen are displayed. In a possible implementation manner, the virtual interactive interface may also include virtual image information. For example, the virtual image information may include a text editing interface, the text editing interface may include text content recognized according to the text image included in the scene image, and virtual keys that can be used to edit the text content Wait. The user can use a stylus pen to perform processing such as drawing, signing, text input or editing content in the virtual interactive interface.

The display position of the virtual interactive interface may be a predetermined position in the scene image. For example, as shown in FIG. 3, the virtual interactive interface may be superimposed and displayed in a predetermined area at the lower right corner of the preset scene image. Of course, the virtual interactive interface can be determined according to the plane information detected in the scene image.

As shown in FIG. 4, it can be determined whether the flat area in the current scene meets the predetermined flat area requirements according to the preset flat area requirements. If there is a plane area in the current scene that meets the requirements of the predetermined plane area, the corresponding virtual interactive interface can be directly generated according to the plane area in the current scene. For example, in FIG. 4, if it is detected that the desktop 303 in the image of the current scene meets the requirements of the predetermined plane area, the position of the virtual interactive interface can be determined according to the position of the desktop in the image of the current scene. Alternatively, the shape of the virtual interaction may also be determined according to the shape of the desktop.

The preset requirements for the plane area may include one or more of the limited conditions such as the plane size range, the plane orientation, and the distance between the plane and the camera or the user. The size of the plane may include the minimum size requirement of the set plane. The plane orientation may include upwards or toward the user, or the like, or the plane orientation may include the tilt angle range of the plane. For example, the inclination angle range may include a range from a horizontal angle to an angle perpendicular to the horizontal plane. In order to facilitate the user to interact with the plane in the scene image through the stylus, the distance of the plane from the camera or the user may be smaller than the first set distance value. For example, the limiting conditions may include the size of the plane, the orientation of the plane, and the distance between the plane and the camera, and a better plane can be obtained through the combination of plane size screening, plane orientation screening, and the distance between the plane and the camera.

In an implementation scenario, as shown in FIG. 5, when the scene includes multiple planes, and the multiple plane areas all meet the requirements of the virtual interactive interface. Then, according to the predetermined preferred writing position of the stylus, a preferred plane among the multiple planes can be selected as the virtual interactive interface. It can be selected according to the distance between the plane and the better writing position. When the plane is closer to the better writing position, the preferred choice is. For example, FIG. 4 includes a flat area A and a flat area B that meet the requirements of a preset virtual interactive interface. The flat area A is closer to the preset better writing position N, and the flat area A is selected as the virtual interactive interface.

In a possible implementation manner, as shown in FIG. 6, the preferred writing position may also be a preset preferred writing area. When the virtual interactive interface is determined according to the preferred writing area. A number of candidate planes can be determined according to the requirements of the virtual interactive interface, and then the area of the intersection area between the multiple candidate planes and the preferred writing area can be calculated, and the candidate plane with the largest intersection area is selected as the virtual interactive interface . For example, in FIG. 6, it is detected that the current scene includes a flat area C and a flat area D. The flat area C, the flat area D, and the better writing area M are calculated for the intersection area, and the plane D with a larger intersection area is selected as the virtual interactive interface. .

Wherein, the preferred writing position of the stylus can be determined by receiving a user designation, or, according to the characteristics of the human body’s physiological structure, when the user’s arm and elbow are at a predetermined angle, the user’s holding The position of the tip of the stylus is determined as the preferred writing position. For example, the state of the user's holding a stylus can be collected through a camera, and when the arm and elbow are detected at a predetermined angle, the position of the pen tip at that moment can be recorded as the preferred writing position of the stylus. The preferred writing area can be determined by the preferred writing position. For example, the preferred writing area may be a rectangular area, an elliptical area or an area of other shapes with the preferred writing position as the center.

When the electronic device includes a head-mounted display device, as shown in the schematic layout diagram of the head-mounted display device as shown in FIG. 7, the display unit may include a first display unit 211 and a second display unit 212, which can be used to display The constructed video of the 3D scene or the image including the virtual interactive interface, the user can watch the generated 3D scene image by wearing the head-mounted display device. A visual perception unit 22 and a depth sensing unit 23 can be provided in the middle of the display unit and the second display unit, or a light supplement unit 24 and the like can be further included.

Through the display device, a virtual interactive interface can be displayed on the interactive image of the current scene, and the data input by the user can be displayed on the virtual interactive interface.

In one implementation, the user can write on any plane in the scene with a stylus. The stylus detects the relative displacement data of the stylus tip through the relative motion sensing unit, and determines the writing trajectory of the tip according to the writing. The trajectory determines the user's writing data, including drawing trajectory, text content or virtual key operation, etc. The written data is displayed on the virtual interactive interface showing a fixed position shown in FIG. 3, or in response to a user's virtual key operation, for example, in response to a virtual key deletion operation.

In one implementation, when the stylus is in the empty state, the camera can detect and track the identification feature of the stylus tip to determine the position of the stylus tip. By detecting and tracking the position of the pen tip of the stylus in the air, the initial position of the pen tip in the writing state can be determined. As shown in FIG. 4, the virtual interactive interface and the plane of the object in the scene have the same position in the image of the current scene, that is, the interactive image. By detecting the initial position of the stylus in the writing state, combined with the relative movement information of the pen tip, the corresponding writing trajectory can be displayed in the corresponding position of the virtual interactive interface, so that the user can view the writing trajectory at the writing position through the head-mounted display device , So as to better match the user's usual writing habits, and enhance the user's writing experience with the stylus. Or according to the initial position of the writing state, the click operation triggered by the stylus on the function button corresponding to the position is received.

Among them, the information written by the user using the stylus pen on the object plane in the scene may include text, images, editing instructions, and so on.

For example, when the user writes characters on the object plane, the track written by the user can be recorded, and the characters corresponding to the track can be recognized, and the recognized characters can be displayed in the virtual interactive interface.

Alternatively, the virtual interactive interface includes text and keys for editing the text. The user can trigger the keys in the virtual interactive interface through a stylus to realize the editing process of the corresponding text, and display the response information of the editing process through the virtual interactive interface. . In the schematic diagram of the virtual interactive interface shown in FIG. 8, the user adds a comment 802 to the text included in the virtual interactive interface through the "comment" button 801. When the text in the virtual interactive interface is annotated through the "Annotation" button, the user can use the stylus to move to the position to be annotated, and click on the position to be annotated to keep the editing cursor in a blinking state at the position to be annotated . Move the stylus to the position corresponding to the comment button 801, and the comment instruction corresponding to the comment button can be triggered by a trigger method such as clicking, and the comment 802 is generated at the cursor position. When the comment is active, the content of the comment added by the user can be displayed in the comment box.

Alternatively, the virtual interactive interface can be used to display the signature data input by the user to the file in the current scene, or to copy, paste, cut, and other operations on the text in the current scene. In a possible implementation manner, it is also possible to perform text recognition on text media in the current scene, such as text images corresponding to books and paper, to obtain text content corresponding to the text images. After editing the text content, the edited text is saved, for example, the modification to the text is saved, or the annotation or signature to the text is saved.

The first processing unit 150 can be used to process the collected video images and the received sensor information, including acquiring the depth information of the object in the current scene, and performing SLAM based on the depth information of the object and the image of the current scene in combination with camera parameters Synchronize positioning and mapping, and determine the absolute position of the pen tip or fingertip according to the characteristic information of the stylus or fingertip in the image.

Alternatively, the first processing unit 150 or the second processing unit 230 is configured to determine the accurate position of the pen tip of the stylus according to the relative displacement data of the pen tip collected by the stylus, and according to the absolute position and the relative displacement data, so as to Accurately calculate the data corresponding to the trajectory written by the stylus.

The first processing unit 150 may also include performing text recognition on the text medium in the current scene, such as text in a book or paper, to obtain text content that can be edited, combining with the editing information input by the stylus, and recognizing the recognized text. The content is edited. For example, when a text medium such as a book file or paper is detected in the current scene, when it is detected that the stylus is in a pressed writing state, and the pressed position is in the area where the text medium is located, the image of the text medium can be written. Recognition processing, acquiring the text information included in the text media, and editing the text media according to the data processing information written by the stylus, including modifying text content, adding annotation information, copying selection, translation, or in the current scene Trigger instructions for the control buttons of the virtual interactive interface superimposed on it.

Wherein, the text medium detection may perform comparison detection in the collected scene image according to preset text medium characteristics to obtain the text medium included in the scene image.

The first processing unit 150 or the second processing unit 230 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and runs or executes software programs and/or modules stored in the memory 120 , And call the data stored in the memory 120 to execute various functions of the electronic device and process data, so as to monitor the electronic device as a whole. Optionally, the first processing unit 150 or the second processing unit 230 may include one or more processing units; preferably, the first processing unit 150 may integrate an application processor and a modem processor, where the application processor mainly Processing the operating system, user interface and application programs, etc., the modem processor mainly deals with wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the first processing unit 150.

The first storage unit 160 or the second storage unit 240 can be used to store software programs and modules. The first processing unit 150 runs the software programs and modules stored in the first storage unit 160, and the second calculation processing unit 230 runs The software programs and modules stored in the second storage unit 240 execute various functional applications and data processing of the electronic device. The first storage unit 160 or the second storage unit 240 may mainly include a storage program area and a storage data area, where the storage program area may store an operating system and an application program required by at least one function (such as a sound playback function, an image playback function). Functions, etc.), etc.; the data storage area can store data created based on the use of electronic devices (such as video images, edited electronic text or images, etc.), etc. In addition, the first storage unit 160 or the second storage unit 240 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. .

The first power supply unit 170 or the second power supply unit 250 may be a battery. Preferably, the first power supply unit 170 or the second power supply unit 250 may be logically connected to the computing processing unit through a power management system, thereby through the power management system Realize functions such as managing charging, discharging, and power consumption management.

The relative motion sensing unit 220 may include a pressure sensing unit 2201 and a handwriting trajectory information sensing unit 2202. The pressure sensing unit 2201 can detect whether the stylus is in a writing state. According to the pressure sensing unit 2201 detecting the pressure between the pen tip and the writing plane, when the pressure is greater than a predetermined value, the stylus is considered to be in a writing state. The handwriting trajectory information sensing unit 2202 may include a laser interference unit and/or an inertial sensing unit. The relative displacement information of the pen tip is obtained through the laser interference unit, and the magnitude and direction changes of the acceleration of the pen tip are sensed by the inertial sensing unit. Changes in size and direction determine the movement trajectory of the pen tip. Alternatively, the handwritten trajectory information sensing unit may also include a displacement sensing sensor such as a camera, which detects and perceives changes in the relative position of the pen tip through a change in the image collected by the camera set by the pen tip.

The inertial sensing unit may be an acceleration sensor, a gyroscope, etc., for detecting changes in the magnitude and direction of the acceleration of the pen tip. The inertial sensing unit may include a plurality of units, which are respectively arranged at different positions of the pen tip. For example, an inertial sensing unit can be provided at the tip and the end of the pen, and the change in the posture of the stylus can be determined by the difference in acceleration sensed by the inertial sensing unit.

In an implementation manner, the tip of the stylus is also provided with a tip feature. The feature of the pen tip can be a special color mark, or a mark such as infrared reflection or fluorescence. When the stylus is provided with the nib feature, when the camera captures an image of the current scene including the stylus, the nib feature is detected on the image of the current scene, and the nib is determined based on the nib feature detected in the image of the current scene The absolute position in the current scene. When the writing plane of the stylus is consistent with the position of the virtual interactive interface, according to the determined absolute position of the pen tip in the current scene, the position of the pen tip in the virtual interactive interface can be effectively determined to realize the writing position and content of the stylus pen Matching of the display position, thereby better improving the user’s writing experience.

It can be understood that the electronic device described in the present application is not limited to this, and may also include other components not listed. For example, the stylus 2 may also include a display screen, through which the writing status of the stylus pen can be displayed, or information such as time can also be displayed. Or, the second display unit may be a touch screen, and the sensitivity of the stylus to the writing state detection can be adjusted through the second display unit. For example, the pressure threshold set by the stylus is F1, when the pressure sensor of the stylus currently detects that the pressure of the tip of the stylus is F2, if F2 is greater than F1, it is considered that the stylus is currently in the writing state. The currently detected pressure of the pen tip of the stylus is F3. If F3 is less than F1, it is considered that the stylus is currently empty. The size of the pressure threshold can be adjusted through the second display unit. When the pressure threshold is increased, the sensitivity of writing state switching decreases, and a greater pressure is required to trigger the writing state. When the pressure threshold is adjusted smaller, the sensitivity of writing state switching increases, and a smaller pressure can trigger the writing state.

When a user uses an electronic device such as a mixed reality system to input data information, in order to solve the problem of handwriting input data, the image captured by the camera on the electronic device is detected by a visual algorithm to detect the position of the pen tip to correct the problem of inaccurate movement trajectory of the stylus, and The need to rely on the handwriting board for input makes handwriting input inconvenient. This application proposes an electronic device interaction method as shown in FIG. 9 as an example and not a limitation. The method can be applied to the above electronic device.

In step S901, the handwriting interactive device displays an interactive image.

The interactive image may be an image of the current scene acquired by a visual perception module, or may be other multimedia images to be played, such as video images, PPT images, and so on.

Wherein, when the interactive image is an image of the current scene, in order to accurately obtain the cursor position corresponding to the pen tip on the virtual interactive interface, real-time positioning and mapping SLAM can be performed on the current scene, so as to facilitate the reconstruction of the three-dimensional scene model, Determine the position of the detected pen tip relative to the object in the scene when the pen is raised, and determine the handwriting interaction area or the interaction position in the handwriting interaction area corresponding to the position of the pen tip according to the position change.

Wherein, the handwriting interaction area may be an area in the current scene touched by the stylus pen. The handwriting interaction area is usually a flat area. For example, areas such as desktops, walls, or other flat surfaces.

The following describes the scene reconstruction.

1.1 Scene image acquisition

The image of the current scene can be acquired through a visible light camera, such as the first camera shown in FIG. 1. In a possible implementation manner, multiple cameras can also be used to obtain multiple images of the current scene, and the multiple images can be spliced according to the angle and position of the cameras corresponding to the multiple images, or according to the content of the images, to obtain more information. For a comprehensive image of the current scene. The image can be a video or other multimedia format.

In one implementation, as shown in Figure 10, the main device is provided with a visible light camera A and a visible light camera B. The visible light camera A obtains a first video image P1, and the visible light camera B obtains a second video image P2. Set the position and angle relationship of the visible light camera A and the visible light camera B, or splicing the first video image P1 and the second video image P2 according to the image content of the first video image P1 and the second video image P2, The spliced video image P is obtained, or the spliced video image can also be enhanced.

1.2 Obtaining scene depth information

When acquiring the depth information of the current scene, the depth information corresponding to the object in the image can be determined based on the images taken by two or more cameras, or the depth information between the object in the current scene and the main device can be acquired based on the depth sensor. The distance between.

Using the images acquired by two or more cameras, the depth information of the objects in the image can be determined based on the principle of triangulation ranging. As shown in FIG. 11, when two cameras with the same parameters are located on the same plane, and the focal length f of the two cameras and the center distance T of the two cameras are known in advance. When acquiring the depth information of the object in the image, the disparity (Xr-Xt) of the object in the two images can be determined according to the position of the object in the captured image. The depth information of the object is Z, as shown in Figure 11. The triangle similarity principle of can be obtained: [T-(Xr-Xt)]/T=(Zf)/Z, which can be solved to obtain:

Z=f*T/(Xr-Xt),

Due to the focal length f, the center distance B between the two cameras is known in advance. By detecting the position of a feature point in the image in the two images, the parallax (Xr-Xt) can be determined, and thus the depth information Z can be calculated.

When more than two cameras determine the depth information, you can select the images taken by any two of the cameras to calculate the depth information, or you can also calculate multiple depth information in the map, and average according to multiple depth information, etc. Optimization method to determine the best depth information.

When acquiring the depth information of the object, the distance measurement module or system such as the time-of-flight ranging camera, structured light ranging system, radar ranging system, ultrasonic sensor, etc. in the depth sensing unit can also be used to obtain the information in the scene. Depth information of the object.

1.3 Match the object and depth information in the image

When the depth sensing unit is two or more cameras, the depth information of the object in the image is calculated according to the position of the same object acquired by the camera in the image captured by the two or more cameras. The calculated depth information is directly matched with the position in the image.

Or, when the depth sensing unit is another distance measuring unit, the corresponding position of the object in the image can be determined according to the position of the measured object. Or, the depth information corresponding to the object in the image can be determined according to the feature information of the distance information measured by the distance measuring unit, including the change feature of the distance, and the matching relationship with the object in the image. Based on the determined depth information of the object, the coordinates of the image taken by the camera can be obtained and converted into a coordinate transfer matrix of the coordinate position of the object in the current scene in the world coordinate system.

According to the camera parameter information of the camera, the coordinate conversion matrix between the coordinates in the image coordinate system of the camera image and the world coordinate system of the three-dimensional space can be obtained. According to the calculated depth information, combined with the coordinate conversion matrix, the coordinate position of the coordinate position of the image captured by the camera in the world coordinate system can be obtained.

1.4 Reconstruction of the current scene

According to the acquired depth information, the surface shape information of the object in the current scene can be determined, and the object in the current scene can be reconstructed according to the shape information. According to the acquired image information, the feature points in the image can be detected and the feature information in the image can be obtained.

The feature information in the image may be a feature point used to describe an object in the image, including, for example, a corner point describing the object.

In an implementation manner, the feature information detection may further include detecting a plane in the scene, and determining the plane area included in the scene. The inner edge detection may be further performed on the plane area. Among them, the inner edge detection refers to the detection of the edge line feature located in the detected plane area, and it is judged that the inner edge area exists.

As shown in FIG. 12, in the image S of the current scene collected by the camera, according to plane feature detection, including, for example, depth sampling detection, the plane area S1 of the desktop is obtained. The inner edge detection is performed on the plane area S1 to obtain the inner edge area determined by the inner edge, that is, the plane area S2. It can be determined whether the flat area S2 includes a square interactive area, or whether the flat area S2 includes a text medium such as a book or the like. Wherein, the square interaction area can be detected according to the size feature or shape feature of the predetermined square interaction area.

When it is recognized that the inner edge area, that is, the flat area S2, includes the text medium, the image corresponding to the text medium can be subjected to text recognition processing to obtain the text content included in the text medium, and all the text content can be displayed in the virtual interactive interface. The recognized text content is convenient for the user to edit the text content in the text medium. Including operations such as modifying the content in the text medium, copying the content in the text medium, or labeling and remarks. Or, according to the content in the selected text medium, information related to the selected text medium content can be obtained, including, for example, obtaining translation information of the selected content, or obtaining other related search information of the selected content, etc.

Based on the current scene reconstructed in 1.3, it may further include step S902. The handwriting interaction device acquires the first operation of the stylus according to the movement information of the stylus, and the movement information includes the relative movement information of the stylus.

The first operation may be a writing operation or a clicking operation in the writing state, or may also include a moving operation when the stylus is in the empty state.

The accurate relative position detection of the relative motion sensing unit can be used to obtain the trajectory information of the stylus when the stylus is in the writing state, that is, the writing trajectory of the stylus. The relative motion sensing unit may include one or more of sensing devices such as a gyroscope, an inertial sensor, or an acceleration sensor.

In an implementation manner, the tip of the stylus can also be positioned, so that the change of the position of the writing content of the stylus can be determined according to the change of the position of the stylus in space. The position of the pen tip of the stylus in the image can be detected, and the stylus pen tip can be located and tracked according to the position of the detected pen tip in the image, which can specifically include:

2.1 Pen tip feature recognition

Find the position of the pen tip in the acquired colored image according to the pre-set characteristics of the pen tip, including the special color feature set by the pen tip, or the fluorescent feature set on the pen tip, or according to the reflection of the infrared reflective material set on the pen tip in advance Features, find the position of the pen tip in the infrared image.

2.2 Acquisition of tip depth information

According to the searched pen tip position, the depth information corresponding to the pen tip position obtained in advance according to the depth sensing unit can be used. When the user moves while wearing the main device, such as wearing a head-mounted display device, the depth information corresponding to the position of the pen tip is calculated in real time according to the real-time image obtained by the camera.

2.3 Get the space position of the pen tip

Based on the coordinate conversion matrix of the camera determined in step 1, the spatial position of the pen tip can be calculated according to the position of the pen tip in the image and the depth information corresponding to the position.

In the schematic diagram for determining the position of the pen tip as shown in Figure 13, the image includes the uv coordinate system that determines the position of the pixel in the image, the camera coordinate system XcYcZc determined by the camera, and the world coordinate system XYZ, where the camera coordinate system is The distance between the origin of and the imaging plane is determined according to the camera parameters.

In the image taken by the camera, after feature detection, it is determined that the position of the pen tip P of the stylus in the image is point A. When the depth sensing unit is a dual camera, the camera center distance between the dual cameras, the focal length of the dual cameras, and the parallax of the images captured by the pen tip P of the stylus on the two cameras can be combined as shown in FIG. 11 Based on the triangle similarity principle, the distance of the pen tip P of the stylus relative to the camera is calculated, that is, the depth information of the pen tip P.

Alternatively, the distance between the pen tip P and the camera can also be measured by a depth sensor, such as a radar, a time-of-flight ranging camera, a structured light ranging system, and an ultrasonic sensor.

According to the distance between the pen tip and the camera detected by the depth sensing unit, that is, the depth information corresponding to the image point A corresponding to the pen tip in the image captured by the camera, the point P in the current scene can be uniquely determined. According to the coordinate conversion matrix from the image coordinate system to the world coordinate system determined in step 1, the pen point is determined by combining the position of the pen tip in the image (point A) and the depth information corresponding to the pen tip position (point A) The world coordinate corresponding to the position in the world coordinate system. And according to the changes of the collected images, the world coordinates corresponding to the pen tip can be determined in real time, and the pen tip can be tracked in real time.

In step S903, the electronic device displays a virtual interactive interface on the interactive image, and responds to the first operation on the virtual interactive interface.

A virtual interactive interface can be generated at any position on the interactive image, or a virtual interactive interface can be generated for a fixed position on the interactive image, or one of the planar areas included in the interactive image can be selected Virtual interactive interface.

When the pen tip is in the pen-up state, the position of the pen tip in the virtual interactive interface can be determined according to the spatial position of the pen tip and the generated three-dimensional scene model. When the pen tip is in the pen-down state, according to the stylus The relative motion sensing unit acquires the handwriting information of the stylus in the virtual interactive interface.

Wherein, the relative motion sensing unit may be an inertial sensor or an activation interferometer or other sensing equipment.

In order to display the acquired operation data corresponding to the stylus pen, it is necessary to generate a virtual interactive interface in the electronic device in advance. Depending on the type of electronic device, the method of generating the virtual interactive interface may also be different.

When the electronic device is a virtual reality VR device, the virtual interaction interface can be directly displayed on the display unit of the virtual reality VR device, and the position in the virtual interaction interface and the position in the current scene can be established. Correspondence. According to the absolute position of the stylus in the current scene, the position of the cursor corresponding to the stylus on the virtual interaction interface or the trajectory of the stylus is determined.

The virtual interaction interface displayed in the virtual reality VR device may be a fixed area in a virtual screen, or may also be a plane area that moves with the line of sight and is located at a predetermined distance within the range of the line of sight.

When the electronic device is an augmented reality AR device or a mixed reality MR device, it is necessary to match the current scene with the virtual interactive interface, which may specifically include:

3.1 Determine the current scene image displayed

The current scene image displayed on the display unit may be an image of the current scene directly transmitted through the semi-transparent display unit, or may also be a display unit, such as an image of the current scene captured by a camera. When the display unit displays the image of the current scene, the displayed view of the current scene image is basically the same as the view through the semi-transparent display unit, so that after matching the virtual interaction interface, the user’s experience in the virtual interaction is improved. The coordination of the operations in the interface.

3.2 Display virtual interactive interface

While displaying the current scene image captured in real time or through the current scene image, the display unit also displays a virtual interaction interface in the current scene image to facilitate the user to interact with the current scene image.

The setting of the position of the virtual interactive interface may be fixed in a fixed area in the virtual screen, or may be a plane range that moves with the line of sight and is located at a predetermined distance within the range of the line of sight.

Alternatively, the virtual interaction interface may correspond to the position of the handwriting interaction area in the current scene image. According to the position of the handwriting interaction area in the real space, the handwriting interaction area in the real scene or real-time image signal is used as the virtual interaction interface. When the position of the writing interaction area in the image changes, the position displayed on the display unit of the virtual interaction interface also changes accordingly.

When the current scene image includes multiple planes, the handwriting interaction area in the current scene image may be determined according to the position of the stylus, and the virtual interaction interface may be adjusted according to the determined handwriting interaction area.

For example, the image of the current scene includes plane X and plane Y. When the user inputs data on plane X with a stylus pen or fingertip, such as inputting formulas or sketches, the handwriting interaction area is displayed on plane X, When the user's stylus or fingertip inputs data on the plane Y, the position of the virtual interaction interface is changed, and the corresponding virtual interaction interface is generated on the plane Y. In an implementation manner, the virtual interactive interface generated on the plane Y can retain the previously input data, thereby enabling user input to reduce environmental constraints and greatly improve the convenience of input.

In an implementation manner, when the current scene includes a real text medium and the stylus interacts with the real text medium, the virtual interaction interface may cover an area of the real text medium. It can electronically identify the real text medium (the content of the document, and display the electronic content in the corresponding position of the real text medium, and the position of the recognized content can be compared with the position of the content in the real text medium One-to-one correspondence, the real text medium in the current scene can be directly displayed on the display unit, or the electronic text content can be displayed. The virtual interaction interface can also include the physical range of the handwriting interaction area. The handwriting interaction The physical range of the area can be determined according to the visual features such as the edge of the scene. For example, the handwriting interaction area can be the edge corresponding to the real text medium, that is, the area determined by the paper.

In a possible implementation manner, when the virtual interaction interface may be a fixed area in the display unit, the virtual interaction interface is displayed at a fixed position in the virtual interaction interface. The virtual interactive interface may include preset keys or buttons to facilitate the user in certain fixed places, for example, the user can complete operations such as live broadcast, formula input, and sketch drawing through the electronic device described in this application.

Wherein, when detecting the state of the stylus pen in the present application, it can be determined by the pressure sensing unit provided at the position of the tip of the stylus pen. When the pressure sensed by the pressure sensing unit is greater than a predetermined value, it is determined that the stylus pen is in a writing state, and when the pressure sensed by the pressure sensing unit is less than or equal to a predetermined value, it is determined that the stylus pen is raised. Pen status.

Alternatively, the state of the stylus pen or the fingertip can also be determined by the acquired absolute position of the pen tip or the fingertip and the position between the handwriting interaction planes in the reconstructed current scene.

When the stylus or fingertip is in the pen-up state, according to the calculated absolute position determined by the stylus pen tip or the fingertip of the finger, and the reconstructed current scene image, it is determined that the fingertip or the pen tip is at The position in the current scene image. According to the position of the fingertip or pen tip in the current scene image, the position of the trajectory of the operation input by the user can be determined. The content in the current scene according to the data corresponding to the position of the track, or the content in the virtual interactive interface corresponding to the operation input by the user.

When the stylus is in the pen-down writing state, due to the occlusion of the hand or the stylus, the image collected by the camera may not show the position of the fingertip or the pen tip, or the trajectory of the writing occupies a larger range in the image. If it is small, it will affect the accuracy of the recognized pen tip or the track content written by the fingertip. In order to overcome this problem, when the pressure sensing unit or visual image detects that the stylus is in the pen-down state, the relative movement sensing unit described in the embodiments of the present application can obtain the relative movement track information of the stylus.

Wherein, when the relative displacement information of the stylus is acquired by the relative motion sensing unit, the relative displacement information of the pen tip can be acquired by the handwriting trajectory information sensing unit. For example, the relative displacement information of the pen tip may be obtained through the laser interference unit in the handwriting trajectory sensing unit, and the handwriting of the stylus can be determined according to the relative displacement. Or, the change in the magnitude and direction of the acceleration of the pen tip of the stylus pen is sensed by an inertial sensing unit to determine the relative displacement information corresponding to the pen tip.

Or the handwriting trajectory sensing unit may include a camera, and the distance and direction of the pen tip movement is determined according to the change of the picture taken by the camera.

When acquiring the relative movement trajectory information, the pen tip pressure collected by the pressure sensing unit can also be acquired, and the line thickness of the relative movement trajectory information is determined according to the magnitude of the pen tip pressure, so as to obtain a more accurate relative movement trajectory information. Movement track information.

In the inertial sensing unit provided by the stylus, the inertial sensing unit may include an acceleration sensor and/or a gyroscope. The inertial sensing unit may include a plurality of them, and they may be arranged on different parts of the stylus, for example, they may be arranged at the positions of the pen tip and the end of the pen respectively, and the magnitude and magnitude of acceleration of the tip of the stylus and the end of the pen may be obtained by the inertial sensing unit. The direction is to determine the posture information of the stylus based on the magnitude of the acceleration and the change in the direction. The determined posture information can be adjusted to the posture of the stylus in the virtual interactive interface, so that the user can get a more realistic writing experience.

In order to further improve the user's writing experience, before acquiring the writing handwriting of the stylus pen, the step of calibrating the relative movement direction of the stylus pen may also be included. The guide line can be generated in the area where the virtual interactive interface and the real image are superimposed. The guide line may be a straight line or a curve of other shapes. When the user draws the guide line in the handwriting interaction area with the stylus, the visual perception module obtains the corresponding first movement track during the drawing process of the stylus, and the relative movement sensing unit obtains the nib of the stylus. The relative displacement information determines the second movement trajectory. By comparing the difference between the first movement trajectory and the second movement trajectory, the relative movement direction is calibrated and adjusted.

For example, the position of the pen tip in the image is acquired through the visual perception module, the depth information of the pen tip is acquired according to the visual perception module or the depth sensing unit, and the position of the pen tip in the image and the depth information of the pen tip are used to determine that the pen tip is in the current scene According to the obtained video image, track the spatial position of the pen tip in the current scene to obtain the absolute movement track of the pen tip, that is, the first movement track. The second movement track of the pen tip is acquired according to the movement sensing unit, and the difference between the first movement track and the second movement track is compared. For example, if the collected first motion track is a straight line with an inclination angle of A, and the second motion track is a straight line with an inclination angle of B, the tilt angle A of the motion sensing unit is adjusted to the adjustment range |B-A|.

In addition, in order to further improve the perception accuracy of the handwriting trajectory information perception unit, a distance sensor may also be provided on the stylus pen. The distance between the stylus pen and the handwriting interaction area is detected by the distance sensor, and when the distance is less than a preset value, the position of the pen tip is measured by the laser interference unit and/or the inertial sensing unit provided on the stylus pen The fine adjustment improves the accuracy of visually positioning the pen tip position, and obtains more accurate relative motion information.

In a possible implementation of the present application, the real text medium in the current scene can be identified, and the content of the text corresponding to the identified text medium can be saved. Or you can receive the user's editing information and save the edited content.

When editing the content of the text, the edited text can be determined according to the position of the stylus. For example, editing operations such as text selection, copying, retrieval, or translation can be realized according to the pen style of the stylus, the keys of the stylus, gestures or voice commands, or the translated content can be played through the speaker.

In a possible implementation of this application, the scene image information and the handwriting interaction process can be saved or transmitted to the network, which can facilitate the sharing of the interaction process with other users.

Based on the electronic equipment and the electronic equipment interaction method described in this application, it can be widely used in education, office, entertainment and other fields. The convenience and input accuracy of handwriting input can be improved, and the operating experience can be greatly improved. Give a brief example to illustrate.

For example, in an office scenario, a user can wear the electronic device described in this application when reading a paper document, such as a book or other paper containing text. The electronic device may include a head-mounted display device and a stylus pen. The image 1402 including the paper file 1401 as shown in FIG. 14 is captured by the visual perception module in the head-mounted display device, and the depth information corresponding to the image is acquired by the depth sensing unit to construct the 3D corresponding to the captured image. Model. The inner edge of the plane in the 3D model is detected, and the text medium of the paper document 1401 included in the plane 1403 in FIG. 14 is identified.

According to the text medium recognized in Figure 14, the image area corresponding to the text medium can be intercepted, and the text content in the text medium can be recognized by OCR recognition, and the text content in the text medium can be generated as shown in Figure 15 The virtual interactive interface 151 is shown, and the position of the virtual interactive interface is determined according to the position of the text medium in the image (not limited to this, the virtual interactive interface can also be fixed in the image as needed. The method of a position, or the area fixed at a predetermined distance in the direction of the line of sight).

In order to improve the convenience for users to annotate text, as shown in FIG. 15, the virtual interactive interface also includes a toolbar 152. The toolbar includes selection boxes 153 of different colors. The user can move the stylus , Select different colors to add different background colors to the text content, or add different annotations to the text through the selected color, and generate annotations 154 and so on. Of course, the content of the toolbar is not limited to this, and can also include editing tools such as copy, paste, cut, bold, and undo.

When the user edits the virtual interactive interface shown in FIG. 15 through the stylus pen, the position of the pen tip in the image can be determined according to the visual features set by the stylus pen tip, including features such as special color, fluorescence, or reflection. According to the depth information corresponding to the image acquired by the depth sensing unit, the depth information corresponding to the pen tip is determined. According to the position of the pen tip in the image, the depth information of the pen tip, and the coordinate conversion matrix, the spatial position corresponding to the pen tip in the current scene can be calculated.

According to the pressure sensing unit provided on the pen tip of the stylus, or according to the detected spatial position of the pen tip, it is determined whether the stylus is in the pen-up state or the pen-down state.

When the stylus is in the pen-up state, it can be determined that the pen tip corresponds to the cursor position in the virtual interactive interface according to the 3D model corresponding to the constructed image and the position of the pen tip in the 3D model. For example, the vertical line of the plane where the handwriting interaction area is located may be generated according to the spatial position of the pen tip, and the intersection point of the vertical line and the handwriting interaction area is the cursor position corresponding to the pen tip. When the stylus is in the up state, the cursor position corresponding to the tip of the stylus is updated in real time, so that when editing the text, the user can clearly know the content to be edited when the pen is about to be lowered.

A distance sensor can be provided on the tip of the stylus. When it is detected that the distance of the tip from the handwriting interaction area is less than a predetermined value, the position of the tip can be adjusted by the relative motion sensing unit, thereby improving the positioning accuracy of the tip position.

For example, the spatial position of the pen tip that can be determined by the visual perception module and the depth sensing unit corresponds to the cursor position in the virtual interaction interface as M, when the distance between the pen tip and the handwriting interaction area is less than a predetermined value, such as less than 0.5 cm If the relative motion sensing unit determines that the cursor position of the pen tip in the virtual interactive interface is N, then the position of the cursor in the virtual interactive interface can be adjusted by fine-tuning, so as to improve the accuracy of the acquired handwriting . Wherein, the relative motion sensing unit can determine the cursor position corresponding to the pen tip by means of laser interference, inertial sensing unit or camera shooting image.

In addition, in order to reduce the error of the pen tip trajectory detected by the relative motion sensing unit, a calibration button can be generated in the virtual interactive interface, or the calibration button can be set on the stylus. When the user touches the calibration button, the calibration button can be The trigger information of the calibration button generates a calibration line on the virtual interactive interface.

In the device calibration schematic diagram shown in FIG. 16, for the calibration line generated in the virtual interactive interface, the user can view the calibration line through the head-mounted display device. By means of visual perception, the user can draw at the perceived position according to the drawing prompt of the calibration straight line, and according to the position of the perceived straight line. At this time, the absolute movement trajectory of the pen tip, that is, the first trajectory L1, can be obtained through the position of the pen tip captured by the head-mounted display device.

Through the relative motion sensing unit provided by the stylus pen, the motion trajectory of the pen tip can also be obtained, and the second trajectory L2 can be obtained.

According to the difference between the first trajectory 11 and the second trajectory L2, the error occurring in the second trajectory L2 can be calibrated. For example, for the direction deviation degree of the second trajectory, the direction of the direction sensor in the relative motion sensing unit is calibrated.

The relative motion sensing unit can be calibrated according to a predetermined time period, so that the accuracy of the track information detected by the system can be further ensured.

After editing the electronic text content, the edited text content can be saved for easy sharing with other users or convenient for users to view by themselves. For example, view the annotation information for part of the text in the book, and the cross-line information for part of the text in the book. Or, you can perform operations such as translation and reading aloud on the selected text according to the translation, reading and other tools included in the toolbar. According to the different functions implemented, different function labels can be displayed on the virtual interactive interface. When a user triggers a function label, the function button corresponding to the function label is displayed. For example, the edit label includes comments, Delete, copy and other function keys.

Alternatively, as shown in FIG. 17, the virtual interactive interface 172 may display the file to be signed. The contract document may be the image 171 corresponding to the contract document received by the first communication unit or other documents that need to be signed.

Alternatively, the document to be signed can also be a contract paper in the current scene. The image of the document to be signed can be obtained through the visual perception module in the head-mounted display device. After the image corresponding to the document to be signed is obtained, , Receive the user's signature processing, add signature data to the image corresponding to the file to be signed, and send the signed image to other users that need to be signed, so as to conveniently implement a quick and convenient handwriting pen signature operation.

When the document to be signed is the image corresponding to the paper in the current scene, and the position of the virtual interactive interface in the scene image is consistent with the position of the handwriting plane in the scene image, the user can directly The signature position is used for the signature operation. According to the consistency between the display position of the virtual interactive interface and the position of the handwriting plane, the signature information can be accurately generated at the signature of the image of the document to be signed, so that the user can feel the experience of using the stylus to write on real paper, and can Get the image corresponding to the valid signature file.

When the image displayed on the virtual interactive interface is an image of a document to be signed sent by other users, or when the virtual interactive interface is a fixed area in the scene image, the cursor corresponding to the stylus may be displayed on the virtual interactive interface According to the displayed cursor position, the user can move the stylus pen in the handwriting plane, so that the cursor moves to the signature position, completes the signature operation of the file in the virtual interactive interface, and realizes efficient and safe mixed reality office operations.

For another example, the user may wear the electronic device for live explanation. The content that needs to be explained can be played on the display unit in the head-mounted display device. For example, the content that needs to be explained may be a multimedia file, including video, PPT, etc., or the content that needs to be explained may also be the current image collected by the visual perception unit when the user is broadcasting live.

When the display unit plays the multimedia file or displays the current scene, the preset area of the image played by the display unit can be set as a virtual interactive interface, or the entire screen played by the display unit is a virtual interactive interface. In order not to affect the clarity of the played screen, the virtual exchange interface may be set as a transparent layer. When receiving the mark of the stylus in the virtual interactive interface, the corresponding mark information is displayed in the virtual interactive interface, so that the electronic device can realize more convenient explanation and operation of the content displayed by the display unit. . For example, as shown in FIG. 18, when the user uses the electronic device described in this application, the user can draw a mark on the current live broadcast content, that is, the image of the current scene collected by the visual perception unit (the dotted area in the figure) with a stylus. Wait for operations to realize a more vivid explanation.

In a possible implementation, the virtual interactive interface and the image played by the display unit can be synthesized and then sent to other users, or the audio during the explanation can be synthesized and sent to other users, so that other users can get vivid Explain the information.

When marking the content played by the display unit, the writing state of the stylus can be detected. When the stylus is in the pen-up state, the cursor corresponding to the pen tip can be displayed on the virtual interactive interface. When the stylus is in the pen-down state, the writing trajectory of the stylus is acquired according to the relative motion sensing unit of the stylus, The corresponding track content is displayed in the interactive interface. For example, when playing instructional videos on the virtual interactive interface, visual perception is used to determine the cursor position of the stylus in the pen-up state, and relative motion perception is used to determine the corresponding handwriting of the stylus when the pen is down. It is convenient to perform operations such as marking in the handwriting interaction area. . The virtual interactive interface, tag information, and the collected user's voice signal can be shared with other users.

In possible implementation scenarios, formula calculations, drawing drawings, etc. can also be performed through the electronic device described in this application. As shown in FIG. 19, a blank area 191 corresponding to the handwriting interaction area may be displayed in the virtual interactive interface, and a tool 192 corresponding to formula calculation or drawing and drawing may be included around the blank area. Determine the cursor position of the stylus in the pen-up state through visual perception, and determine the corresponding notes of the stylus when the pen is down through the relative motion perception, so that when handwriting input is performed, it is not necessary to rely on a special handwriting tablet, and can effectively improve The input accuracy of the handwriting.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

Corresponding to the electronic device interaction method described in the above embodiment, FIG. 20 shows a structural block diagram of an apparatus provided in an embodiment of the present application. For ease of description, only the parts related to the embodiment of the present application are shown.

Referring to Figure 20, the device includes:

The image display module 2001 is used to display interactive images by electronic devices;

The operation information obtaining module 2002 is configured to obtain the first operation of the stylus pen by the electronic device according to the movement information of the stylus pen, wherein the movement information includes the relative movement information of the stylus pen;

The response module 2003 is configured to display a virtual interactive interface on the interactive image by the electronic device, and respond to the first operation on the virtual interactive interface.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed. Module completion, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist alone physically, or two or more units can be integrated into one unit. The above-mentioned integrated units can be hardware-based Formal realization can also be realized in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the foregoing system, reference may be made to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the above-mentioned embodiments, the description of each embodiment has its own focus. For parts that are not described in detail or recorded in an embodiment, reference may be made to related descriptions of other embodiments.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed device and method may be implemented in other ways. For example, the system embodiment described above is merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components may be It can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the implementation of all or part of the processes in the above-mentioned embodiment methods in the present application can be accomplished by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by the processor, the steps of the foregoing method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate forms. The computer-readable medium may at least include: any entity or device capable of carrying computer program code to the handwriting input device/mixed reality interactive device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random Access memory (RAM, Random Access Memory), electric carrier signal, telecommunications signal, and software distribution medium. For example, U disk, mobile hard disk, floppy disk or CD-ROM, etc. In some jurisdictions, according to legislation and patent practices, computer-readable media cannot be electrical carrier signals and telecommunication signals.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that it can still implement the foregoing The technical solutions recorded in the examples are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in Within the scope of protection of this application.

Claims

An electronic device interaction method, characterized in that, the electronic device interaction method includes:

Electronic equipment displays interactive images;

The electronic device acquires the first operation of the stylus according to the movement information of the stylus, wherein the movement information includes the relative movement information of the stylus;

The electronic device displays a virtual interactive interface on the interactive image, and responds to the first operation on the virtual interactive interface.
The interactive method for an electronic device according to claim 1, wherein the interactive image is an image of a current scene, and the method further comprises:

Obtaining the position of the stylus in the current scene according to the image of the current scene;

According to the position of the stylus in the current scene, the handwriting position of the tip of the stylus in the virtual interactive interface is determined.
The interactive method of electronic equipment according to claim 2, wherein the method comprises:

Obtain the image of the current scene according to the camera;

Or, it also includes acquiring the depth information of the current scene.
The interaction method of an electronic device according to claim 2 or 3, wherein acquiring the position of the stylus in the current scene comprises:

Detect the tip characteristics of the stylus in the image of the current scene, and determine the position of the tip of the stylus in the image of the current scene;

Obtaining the depth information of the pen tip according to the matching relationship between the image of the current scene and the depth information;

The position of the pen tip is determined according to the depth information of the pen tip.
The interactive method for an electronic device according to claim 4, wherein the pen tip feature includes one or more of a color feature, a reflected light feature, or a fluorescent feature.
The interaction method of an electronic device according to claim 2, wherein the determining the handwriting position of the stylus pen tip in the virtual interactive interface according to the position of the stylus pen in the current scene comprises:

When the stylus is in the empty state, obtain the position of the stylus and the position of the writing plane of the current scene, and determine the relative positional relationship of the stylus with respect to the writing plane;

The position of the stylus in the virtual interactive interface is determined according to the relative positional relationship of the stylus relative to the writing plane, combined with a predetermined mapping relationship between the writing plane and the virtual interactive interface.
The interactive method for an electronic device according to claim 1, wherein the interactive image is collected by a camera, or collected by a display device with a predetermined transparency.
The interactive method for an electronic device according to claim 1, wherein the electronic device displaying a virtual interactive interface on the interactive image comprises:

Displaying the virtual interactive interface in a predetermined area in the interactive image;

Or, the virtual interactive interface is determined according to the plane area in the current scene.
The interaction method for electronic equipment according to claim 8, wherein determining the position of the virtual interaction interface according to the plane area in the current scene comprises:

Filter the flat areas in the current scene according to the preset flat area requirements;

The shape and/or position of the virtual interactive interface is determined according to the shape and/or position of the filtered plane area.
The interactive method for an electronic device according to claim 9, wherein the preset requirements for a plane area include one or more of the size range of the plane area, the orientation of the plane area, or the distance of the plane area from the camera.
The interactive method for an electronic device according to claim 9, wherein when there are multiple planar areas that meet the requirements of the planar area, the method further comprises:

According to a predetermined preferred writing position of the stylus, a plane region that more closely matches the preferred writing position is selected from among the plurality of plane regions.
11. The electronic device interaction method according to claim 11, wherein, according to a predetermined preferred writing position of the stylus, selecting a plane region that more closely matches the preferred writing position among the plurality of plane regions comprises:

Respectively acquiring the distances between a plurality of to-be-selected plane areas and the preferred writing position;

Select a flat area that is closer.
11. The electronic device interaction method according to claim 11, wherein, according to a predetermined preferred writing position of the stylus, selecting a plane region that more closely matches the preferred writing position among the plurality of plane regions comprises:

Acquiring a better writing area corresponding to the better writing position;

Respectively acquiring the intersection area of a plurality of to-be-selected plane areas and the preferred writing area;

Select a plane area to be selected with a larger area of intersection.
The interactive method for an electronic device according to claim 1, wherein the electronic device displaying a virtual interactive interface on the interactive image comprises:

Detecting whether the handwriting area corresponding to the current scene of the virtual interactive interface includes a text medium;

If the handwriting area includes text media, an image of the text media is generated on the virtual interactive interface, or text content recognized by the text media is displayed.
The interaction method of electronic equipment according to claim 14, wherein the method further comprises:

Displaying an edit button in the virtual interactive interface;

Responding to the first operation on the virtual interactive interface includes:

When a click operation of the stylus at the corresponding position of the edit button is detected, respond to the corresponding function of the edit button.
The interactive method of electronic equipment according to claim 15, wherein the method further comprises:

Save and/or send the edited image or text content to other users;

Or, select the text content of the virtual interactive interface, and send a request for searching the selected text content to the network;

Receiving and displaying the result corresponding to the request on the virtual interactive interface.
The interactive method of electronic equipment according to claim 1, wherein the method further comprises:

Obtain the image of the pen tip when drawing is performed through the visual perception module, and generate the first trajectory according to the image when drawing is performed;

Generate a second trajectory through relative motion information;

The difference between the first trajectory and the second trajectory is compared, and the relative motion information is calibrated according to the difference.
An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program as follows: The interactive method of electronic equipment according to any one of claims 1 to 17.
A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to realize the interaction of the electronic device according to any one of claims 1 to 17 method.