WO2022021980A1 - Procédé et appareil de commande d'objet virtuel, dispositif électronique et support de stockage - Google Patents

Procédé et appareil de commande d'objet virtuel, dispositif électronique et support de stockage Download PDF

Info

Publication number
WO2022021980A1
WO2022021980A1 PCT/CN2021/091072 CN2021091072W WO2022021980A1 WO 2022021980 A1 WO2022021980 A1 WO 2022021980A1 CN 2021091072 W CN2021091072 W CN 2021091072W WO 2022021980 A1 WO2022021980 A1 WO 2022021980A1
Authority
WO
WIPO (PCT)
Prior art keywords
display position
position information
virtual object
real scene
coordinate system
Prior art date
Application number
PCT/CN2021/091072
Other languages
English (en)
Chinese (zh)
Inventor
李国雄
Original Assignee
北京市商汤科技开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京市商汤科技开发有限公司 filed Critical 北京市商汤科技开发有限公司
Publication of WO2022021980A1 publication Critical patent/WO2022021980A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Definitions

  • the present disclosure relates to the technical field of computer vision, and in particular, to a control method, apparatus, electronic device and storage medium for a virtual object.
  • AR Augmented Reality
  • the embodiments of the present disclosure provide at least one control solution for a virtual object.
  • the present disclosure implements a method for controlling a virtual object, including:
  • the display position of the virtual object in the augmented reality image is controlled.
  • the second display position information of the target key point of the hand in the world coordinate system may be determined based on the first display position information of the target key point of the hand in the device coordinate system and the length information of the arm , by using the length information of the arm, the second display position information of the target key point of the hand in the world coordinate system can be quickly determined, and there is no need to identify a large number of hand skeleton points, so that the recognition efficiency of the target key point of the hand can be improved.
  • the processing efficiency in the process of controlling the virtual object can be improved, and the control effect can be optimized.
  • controlling the display position of the virtual object in the augmented reality image based on the first display position information and the arm length information includes:
  • the display position of the virtual object in the augmented reality image is controlled.
  • the identifying the first display position information of the target key point of the hand in the real scene image in the device coordinate system includes:
  • the identifying the length information of the arm in the real scene image includes:
  • the arm length information in the real scene image is determined.
  • the arm posture type included in the real scene image is determined by image detection, and then the length information of the arm in the real scene image can be quickly determined according to the pre-established mapping relationship.
  • the transforming the first display position information into the second display position information in the world coordinate system based on the first display position information and the arm length information includes:
  • the second display position information is determined based on the third display position information, the camera projection matrix of the image acquisition unit of the terminal device, and the arm length information.
  • normalization processing may be performed on the first display position information of the target key of the hand in the device coordinate system corresponding to different types of terminal devices, so that the augmented reality displayed in different types of terminal devices image, the second display position information of the target key point of the hand of the augmented reality image in the world coordinate system can be quickly determined in a unified manner.
  • the determining the second display position information based on the third display position information, the camera projection matrix of the image acquisition unit of the terminal device, and the arm length information includes: :
  • the camera projection matrix and the length information of the arm determine the fourth display position information of the target key point of the hand in the camera coordinate system
  • the camera projection matrix is a transformation matrix between the standardized device coordinate system and the camera coordinate system
  • the camera extrinsic parameter matrix is a transformation matrix between the world coordinate system and the camera coordinate system.
  • the fourth display position of the target key point of the hand in the camera coordinate system can be quickly determined information, so that the second display position information of the target key point of the hand in the world coordinate system can be quickly determined.
  • controlling the display position of the virtual object in the augmented reality image based on the second display position information includes:
  • the display position of the virtual object in the augmented reality image is controlled to move.
  • the display position of the virtual object can be adjusted based on the position change data of the target key point of the hand in the real scene, so as to achieve the purpose of controlling the display position of the virtual object through the hand.
  • controlling the display position of the virtual object in the augmented reality image based on the second display position information includes:
  • the virtual object in the augmented reality image is controlled to move to the target display position.
  • the display position of the virtual object can be adjusted based on the second display position information of the target key point of the hand in the world coordinate system and the preset relative positional relationship, so as to control the virtual object through the hand. the purpose of the placement.
  • an embodiment of the present disclosure provides a device for controlling a virtual object, including:
  • an acquisition module configured to acquire an augmented reality image in which the real scene image and the virtual object are superimposed
  • an identification module configured to identify the first display position information of the target key point of the hand in the real scene image in the device coordinate system, and to identify the length information of the arm in the real scene image;
  • the control module is configured to control the display position of the virtual object in the augmented reality image based on the first display position information and the length information of the arm.
  • control module is configured to control the display position of the virtual object in the augmented reality image based on the first display position information and the arm length information, including:
  • the display position of the virtual object in the augmented reality image is controlled.
  • the method when the recognition module is configured to recognize the first display position information of the target key point of the hand in the real scene image in the device coordinate system, the method includes:
  • the identifying module when configured to identify the length information of the arm in the real scene image, the method includes:
  • the arm length information in the real scene image is determined.
  • control module is configured to transform the first display position information into a second display in a world coordinate system based on the first display position information and the arm length information
  • location information include:
  • the second display position information is determined based on the third display position information, the camera projection matrix of the image acquisition unit of the terminal device, and the arm length information.
  • control module is configured to determine the second display position based on the third display position information, the camera projection matrix of the image acquisition unit of the terminal device, and the length information of the arm.
  • displaying location information include:
  • the camera projection matrix and the length information of the arm determine the fourth display position information of the target key point of the hand in the camera coordinate system
  • the camera projection matrix is a transformation matrix between the standardized device coordinate system and the camera coordinate system
  • the camera extrinsic parameter matrix is a transformation matrix between the world coordinate system and the camera coordinate system.
  • control module when the control module is configured to control the display position of the virtual object in the augmented reality image based on the second display position information, the control module includes:
  • the display position of the virtual object in the augmented reality image is controlled to move.
  • control module when the control module is configured to control the display position of the virtual object in the augmented reality image based on the second display position information, the control module includes:
  • the virtual object in the augmented reality image is controlled to move to the target display position.
  • embodiments of the present disclosure provide an electronic device, including: a processor, a memory, and a bus, where the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the processing A bus communicates between the processor and the memory, and when the machine-readable instructions are executed by the processor, the steps of the control method according to the first aspect are executed.
  • an embodiment of the present disclosure provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the control method according to the first aspect A step of.
  • FIG. 1 shows a flowchart of a method for controlling a virtual object provided by an embodiment of the present disclosure
  • FIG. 2 shows a flowchart of a method for determining first display position information of a target key point of a hand provided by an embodiment of the present disclosure
  • FIG. 3 shows a schematic diagram of a real scene image including a hand provided by an embodiment of the present disclosure
  • FIG. 4 shows a flowchart of a method for determining arm length information provided by an embodiment of the present disclosure
  • FIG. 5 shows a flowchart of a method for determining second display position information of a target key point of a hand provided by an embodiment of the present disclosure
  • FIG. 6 shows a flowchart of a specific method for determining second display position information of a target key point of a hand provided by an embodiment of the present disclosure
  • FIG. 7( a ) shows a schematic view of a viewing frustum in a camera coordinate system provided by an embodiment of the present disclosure
  • Figure 7(b) shows a schematic diagram of a unit cube in a standardized device coordinate system provided by an embodiment of the present disclosure
  • FIG. 8 shows a flowchart of a method for controlling movement of a virtual object provided by an embodiment of the present disclosure
  • FIG. 9 shows a flowchart of another method for controlling the movement of a virtual object provided by an embodiment of the present disclosure.
  • FIG. 10 shows a schematic structural diagram of a control device for a virtual object provided by an embodiment of the present disclosure
  • FIG. 11 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.
  • Augmented Reality (AR) technology can be applied to an AR device, and the AR device can be any electronic device that can support AR functions, including but not limited to AR glasses, tablet computers, smart phones, and the like.
  • AR device When the AR device is operated in the real scene, the AR scene superimposed by the virtual object and the real scene can be viewed through the AR device.
  • the application of controlling virtual objects based on the hand is very extensive. to change the placement of virtual objects in the AR scene.
  • the position of the center of mass of the hand can be obtained based on the coordinates of each skeleton point of the detected hand, and the display position of the virtual object can be controlled by the movement of the position of the center of mass. Coordinates, and then based on a plurality of coordinates to find the centroid position, the process is cumbersome, and there is a problem of low efficiency when the virtual object is controlled based on the determined centroid position of the hand.
  • the present disclosure provides a control method for a virtual object, which can determine whether the target key point of the hand is in the device coordinate system based on the first display position information and arm length information
  • the second display position information in the world coordinate system by using the arm length information, the second display position information of the target key point of the hand in the world coordinate system can be quickly determined, without the need to identify a large number of hand bone points.
  • the recognition efficiency of the target key points of the hand is improved, and when the display position of the virtual object is controlled based on the target key point of the hand, the processing efficiency in the process of controlling the virtual object can be improved, and the control result can be optimized.
  • the execution subject of the method for controlling a virtual object provided by the embodiment of the present disclosure is generally a computer device with a certain computing capability. Specifically, it may be a terminal device, a server, or other processing device.
  • the terminal device may include AR devices such as a mobile phone, a tablet, and AR glasses, which is not limited herein. AR devices can connect to the server through an app.
  • the control method of the virtual object may be implemented by the processor calling computer-readable instructions stored in the memory.
  • control method includes the following S101-S103:
  • the real scene image can be collected by the image collection unit of the terminal device, and the image collection unit may include a camera set on the terminal device, such as a camera on a smartphone or a tablet, or an external camera, such as an external camera connected to AR glasses. camera.
  • a camera set on the terminal device such as a camera on a smartphone or a tablet
  • an external camera such as an external camera connected to AR glasses. camera.
  • the image acquisition unit of the terminal device After the image acquisition unit of the terminal device collects the real scene image, it can superimpose the real scene image with the virtual object generated in advance with the help of computer graphics technology and visualization technology, so as to obtain an augmented reality image in which the display scene image and the virtual object are superimposed.
  • the augmented reality image can be displayed on the screen of the terminal device.
  • the real scene image is a park
  • the virtual object is a virtual panda superimposed on the park
  • the real scene image is a playground runway
  • the virtual object is a virtual racing car superimposed on the playground runway.
  • S102 Identify the first display position information of the target key point of the hand in the real scene image in the device coordinate system, and identify the length information of the arm in the real scene image.
  • key points of the hand region in the real scene image captured by the image acquisition unit can be identified, the number of key points in the hand region is at least one, and the target key point can be selected from the at least one key point.
  • the center point of the hand may be selected as the target key point, and the key point on the middle finger of the hand may also be selected as the target key point, and the like.
  • the hand contained in the real scene image can be detected based on the pre-trained neural network for target detection, and the detection area containing the hand in the real scene image can be determined, such as containing For the detection frame of the hand, the first display position information of the target key point of the hand in the device coordinate system can be determined based on the position of the detection frame.
  • the device coordinate system may be the coordinate system corresponding to the display screen used to display the augmented reality image in the terminal device, and specifically, a corner of the display screen may be used as the origin of the device coordinate system to display two vertical and intersecting lines on the screen.
  • the edge is used as the coordinate axis of the device coordinate system.
  • a corner of the mobile phone screen can be used as the origin of the device coordinate system, and the long side passing through the corner point can be used as the horizontal axis (x-axis) of the device coordinate system.
  • the short side passing through the corner is used as the vertical axis (y-axis) of the device coordinate system.
  • the device coordinate system can take a corner of the projected lens screen as the origin, and the two lines passing through the origin in the lens screen are perpendicular to each other and intersect.
  • the line is the coordinate system established by the coordinate axis.
  • the real scene image captured by the image acquisition unit also includes an arm
  • the length information of the arm in the real scene image is related to the posture of the arm. For example, the length of the arm in the extended state is longer than that in the retracted state. Therefore, the arm length information contained in the real scene image can be determined by recognizing the posture of the arm in the real scene image.
  • controlling the display position of the virtual object in the augmented reality image based on the first display position information and the arm length information includes: based on the first display position information and the arm length information , transforming the first display position information into the second display position information in the world coordinate system; and controlling the display position of the virtual object in the augmented reality image based on the second display position information.
  • the first display position information corresponding to the target key point of the hand includes the coordinate values of the target key point of the hand along the x-axis and the y-axis respectively in the device coordinate system, because the real scene image is captured by the image acquisition unit, so
  • the second display position information of the target key point of the hand in the world coordinate system may be determined according to the camera projection matrix of the image acquisition unit.
  • a world coordinate system can be pre-built for a real scene.
  • the real scene is an exhibition hall of an exhibition hall.
  • the set position point of the exhibition hall can be used as the origin of the world coordinate system, and three mutually perpendicular straight lines can be used as the world coordinate system.
  • the three coordinate axes of the coordinate system after determining the positive direction of each coordinate axis, the world coordinate system corresponding to the exhibition hall is obtained.
  • the length information of the arm can be used to represent the depth information of the target key point of the hand in the camera coordinate system, and then combined with the first display position information corresponding to the target key point of the hand, it is determined that the target key point of the hand is in the world
  • the second display position information in the coordinate system
  • the initial display position of the virtual object in the augmented reality scene may be determined in a pre-built three-dimensional scene model representing the real scene.
  • the three-dimensional scene model and the real scene are in the same coordinate system, they are presented in a 1:1 ratio.
  • the initial display position of the virtual object in the world coordinate system can be obtained.
  • the determined second display position information is used to adjust the display position of the virtual object in the augmented reality image.
  • the movement of the virtual object in the augmented reality image can be controlled by the movement of the target key point of the hand, or the virtual object can be triggered by the target key point of the hand to control the change of the presentation special effect of the virtual object, such as the virtual object.
  • the special effect of the hand triggering the virtual object can be achieved.
  • the object can start to move with the movement of the target key point of the hand to achieve the purpose of controlling the virtual object through the hand.
  • the second display position information of the target key point of the hand in the world coordinate system may be determined based on the first display position information of the target key point of the hand in the device coordinate system and the length information of the arm , by using the length information of the arm, the second display position information of the target key point of the hand in the world coordinate system can be quickly determined, and there is no need to identify a large number of hand skeleton points, so that the recognition efficiency of the target key point of the hand can be improved.
  • the processing efficiency in the process of controlling the virtual object can be improved, and the control effect can be optimized.
  • S1021 perform hand detection on the real scene image, and determine the detection area including the hand in the real scene image;
  • S1022 Acquire the position coordinates of the target position point of the detection area in the real scene image, and use the position coordinates as the first display position information.
  • the hand detection can be performed on the real scene image based on the pre-trained neural network for target detection, and the detection area including the hand in the real scene image can be obtained, that is, the labeling of the hand position in the real scene image can be obtained.
  • the detection frame of is a schematic diagram of the detected detection frame including the hand.
  • the neural network when the neural network outputs the detection frame containing the hand, it can simultaneously output the position coordinates corresponding to the four corners of the detection frame.
  • the four corner points of the frame include the upper left corner point k1, the upper right corner point k2, the lower left corner point k3 and the lower right corner point k4.
  • the target position point of the detection frame can be determined based on the corresponding position coordinates of the four corner points in the real scene image. , such as the position coordinates of the center point of the detection frame in the real scene image.
  • the straight line equation corresponding to the first diagonal can be determined based on the upper left corner point k1 and the lower right corner point k4, and then the straight line equation corresponding to the second diagonal line can be determined based on the upper right corner point k2 and the lower left corner point k3, and then based on
  • the straight line equation corresponding to the first diagonal line and the straight line equation corresponding to the second diagonal line are used to determine the position coordinates of the center point of the detection frame in the real scene image.
  • the straight line equation corresponding to the first diagonal line and the The intersection point of the line equation corresponding to the second diagonal is used as the position coordinate of the center point of the detection frame in the real scene image.
  • the arm pose category corresponding to the arm contained in the real scene image can be determined.
  • the arm pose type can include three types, such as It includes the arm stretched state, the arm half-extended state and the arm retracted state, and further based on the pre-established mapping relationship between the arm posture type and the arm length information, the arm length information in the real scene image can be determined.
  • the pre-established mapping relationship between the arm posture type and the arm length information may include: when the arm is in an extended state, the corresponding arm length information is 0.65m, and when the arm is in a half-extended state, the corresponding arm length information. It is 0.45m. When the arm is retracted, the corresponding arm length information is 0.2m. If the real scene image is detected and it is determined that the arm posture type in the real scene image is the arm extension state, the arm length information can be quickly determined. is 0.65m.
  • the arm posture type included in the real scene image is determined by image detection, and then the length information of the arm in the real scene image can be quickly determined according to the pre-established mapping relationship.
  • a standardized device coordinate system can be introduced here.
  • the value range of the x-axis and y-axis of the standardized device coordinate system is 0 to 1, so that the first display position information is displayed.
  • the first display position information needs to be normalized.
  • the first display position information can be mapped along the device coordinate system along the The coordinate value of the x-axis is normalized to a value of 0 to 1, and the coordinate value of the first display position information along the y-axis on the device coordinate system is normalized to a value of 0 to 1.
  • the device coordinate system is a coordinate system constructed by the screen of the terminal device
  • the screen size information of the terminal device can be used for normalization.
  • the third display position information of the target key point of the hand in the standardized device coordinate system can be obtained. In this way, the target key point of the hand can be determined in a unified manner in the standardized device coordinate system with different screen sizes.
  • the third display location information is a coordinate system constructed by the screen of the terminal device.
  • the third display position information of the target key point in the standardized device coordinate system P ndc (x ndc , y ndc ):
  • x ndc represents the coordinate value of the target key point of the hand along the x-axis direction under the standardized device coordinate system
  • y ndc represents the coordinate value of the target key point of the hand along the y-axis direction under the standardized device coordinate system
  • x screen Represents the coordinate value of the target key point of the hand along the x-axis in the device coordinate system
  • y screen represents the coordinate value of the target key point of the hand along the y-axis in the device coordinate system
  • W represents the screen of the terminal device in the device coordinate system.
  • L represents the length of the screen of the terminal device along the y-axis in the device coordinate system.
  • the position of the target key point of the hand in the camera coordinate system can be determined based on the third display position information and the camera projection matrix of the image acquisition unit of the terminal device.
  • the fourth display position information wherein the coordinates of the target key point of the hand along the z-axis in the camera coordinate system can be determined by the length information of the arm, and further can be based on the fourth display of the target key point of the hand in the camera coordinate system
  • the position information is used to determine the second display position information of the target key point of the hand in the world coordinate system.
  • normalization processing may be performed on the first display position information of the target key of the hand in the device coordinate system corresponding to different types of terminal devices, so that the augmented reality displayed in different types of terminal devices image, the second display position information of the target key point of the hand of the augmented reality image in the world coordinate system can be quickly determined in a unified manner.
  • the following S10321 to S10322 may be included:
  • S10322 Determine the second display position information based on the fourth display position information and the camera extrinsic parameter matrix used when the image acquisition unit captures the real scene image.
  • the camera projection matrix is the transformation matrix of the standardized device coordinate system and the camera coordinate system
  • the camera extrinsic parameter matrix is the transformation matrix of the world coordinate system and the camera coordinate system.
  • the camera projection matrix M proj can be expressed by the following formula (3):
  • Figure 7(a) and Figure 7(b) are introduced below, and the camera coordinate system corresponding to the image acquisition unit is converted to the standardized device coordinate system.
  • Schematic diagram of conversion Figure 7(a) is the viewing frustum in the camera coordinate system
  • Figure 7(b) is the unit cube in the standardized device coordinate system after the viewing frustum in the camera coordinate system is processed by the camera projection matrix, That is, the camera projection matrix can normalize the frustum.
  • the scene inside the viewing frustum is visible, and the scene outside the viewing frustum is invisible.
  • the viewing frustum includes the far clipping plane ABCD and the near clipping plane EFGH in the z-axis direction perpendicular to the camera coordinate system, where (r, t, n ) represents the coordinate value of point F in the near clipping plane, where r represents the coordinate value of point F along the x-axis direction in the camera coordinate system, t represents the coordinate value of point F along the y-axis direction in the camera coordinate system, and n represents The coordinate value of point F along the z-axis in the camera coordinate system; (l, b, n) represents the coordinates of point H in the near clipping plane, where l represents the coordinate value of point H along the x-axis in the camera coordinate system , b represents the coordinate value of point H along the y-axis direction in the camera coordinate system, n represents the coordinate value of point H along the z-axis direction in the camera coordinate system; f represents the camera focal length corresponding to the image acquisition unit.
  • the common parameters of the camera, the camera field of view fov and the camera aspect ratio aspect can be determined according to the following formulas (4) and (5):
  • the camera projection matrix can also be determined by some intrinsic parameters of the camera field of view, camera aspect ratio, near clipping plane, far clipping plane, and camera focal length.
  • the projection formula (6) is introduced below, and the fourth display position information of the target key point of the hand in the camera coordinate system is determined by formula (6).
  • P cam (x cam , y cam , z cam ) represents the fourth display position information of the target key point of the hand in the camera coordinate system
  • x cam represents the target key point of the hand in the camera coordinate system
  • y cam represents the coordinate value of the target key point of the hand along the y-axis direction in the camera coordinate system
  • z cam represents the coordinate value of the target key point of the hand along the z-axis direction in the camera coordinate system value
  • z cam can be determined by the arm's length information in the camera coordinate system
  • x cam and y cam can be determined by formula (6).
  • the second display position information of the target key point of the hand can be determined by the following formula (7):
  • P world represents the second display position information of the target key point of the hand in the world coordinate system
  • M cam represents the inverse matrix of the camera extrinsic parameter matrix used by the image acquisition unit to capture the real scene image, which specifically represents the camera coordinate system.
  • the camera extrinsic parameter matrix used when the image acquisition unit of the terminal device captures the real scene image may include a translation vector and a rotation matrix when the world coordinate system is converted to the camera coordinate system, which can be captured by the image acquisition unit through the origin of the world coordinate system respectively.
  • the corresponding position information in the camera coordinate system in the real scene image, and the rotation angle of the coordinate axis of the world coordinate system in the camera coordinate system when the image acquisition unit captures the real scene image, or the real scene can be captured by the image acquisition unit
  • the image is represented by the pose data in the world coordinate system.
  • SLAM simultaneous localization and mapping
  • the second display position information of the target key point of the hand can be further determined according to the above formula (7).
  • the fourth display position of the target key point of the hand in the camera coordinate system can be quickly determined information, so that the second display position information of the target key point of the hand in the world coordinate system can be quickly determined.
  • the image acquisition unit of the terminal device may acquire images of the real scene at a set time interval, and determine the second display position information corresponding to the target key point of the hand in each frame of the image of the real scene in the above manner, so that it can be determined.
  • the exhibition hall includes three physical tables, which are respectively denoted as table A, table B and table C, and the virtual object is a virtual vase, and the virtual vase is corresponding to the exhibition hall.
  • the initial position in the world coordinate system is on table A, so the virtual vase on table A can be seen in the augmented reality image, and the virtual vase on table A can be triggered by the target key point of the hand, and then start Move the display position of the virtual vase.
  • the second display position information corresponding to the target key point of the hand in two consecutive real scene images is moved from table A to table C, then the virtual vase can be controlled to move from table A. to table C, thereby presenting an augmented reality image of the virtual vase at position table C.
  • the adjustment process for the display position of the virtual object can be triggered.
  • the adjustment is started for the display position of the virtual object in the augmented reality image.
  • control of dynamic virtual objects can also be completed through the second display position information corresponding to the target key points of the hands in different frames of the real scene images
  • the driving trajectory of the virtual racing car in the augmented reality image is continuously adjusted according to the position change data of the target key points of the hand in the real scene images in different frames.
  • the display position of the virtual object can be adjusted based on the position change data of the target key point of the hand in the real scene, so as to achieve the purpose of controlling the display position of the virtual object through the hand.
  • the preset relative positional relationship between the target key points of the hand and the virtual object may be preset, or may be obtained when the target key points of the hand and the virtual object are acquired in the augmented reality image for the first time. , the initial relative positional relationship between the target key points of the hand and the virtual object.
  • the target display of the virtual object can be determined based on the second display position information of the target key point of the hand obtained in real time in the world coordinate system at the current moment position, and then control the virtual object in the augmented reality image to move to the target display position.
  • the virtual object is a virtual colored brush
  • the augmented reality image can present colored lines corresponding to the movement trajectory, for example, the virtual colored brush can draw pink lines according to the movement trajectory
  • the virtual colored brush can draw pink lines according to the movement trajectory
  • the display position of the virtual object can be adjusted based on the second display position information of the target key point of the hand in the world coordinate system and the preset relative positional relationship, so as to control the virtual object through the hand. the purpose of the placement.
  • the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.
  • the embodiment of the present disclosure also provides a virtual object control device corresponding to the virtual object control method.
  • a virtual object control device corresponding to the virtual object control method.
  • the implementation of the apparatus reference may be made to the implementation of the method, and the repetition will not be repeated.
  • the virtual object control apparatus 1000 includes:
  • an acquisition module 1001 configured to acquire an augmented reality image in which a real scene image and a virtual object are superimposed
  • the identification module 1002 is configured to identify the first display position information of the target key point of the hand in the real scene image in the device coordinate system, and to identify the length information of the arm in the real scene image;
  • the control module 1003 is configured to control the display position of the virtual object in the augmented reality image based on the first display position information and the arm length information.
  • control module 1003 is configured to control the display position of the virtual object in the augmented reality image based on the first display position information and the arm length information, including:
  • the display position of the virtual object in the augmented reality image is controlled.
  • the method includes:
  • the identification module 1002 when configured to identify the length information of the arm in the real scene image, it includes:
  • the arm length information in the real scene image is determined.
  • control module 1003 when configured to transform the first display position information into the second display position information in the world coordinate system based on the first display position information and the arm length information, it includes:
  • the second display position information is determined based on the third display position information, the camera projection matrix of the image acquisition unit of the terminal device, and the arm length information.
  • control module 1003 when the control module 1003 is configured to determine the second display position information based on the third display position information, the camera projection matrix of the image acquisition unit of the terminal device, and the arm length information, it includes:
  • the camera projection matrix and the arm length information determine the fourth display position information of the target key point of the hand in the camera coordinate system
  • the camera projection matrix is the transformation matrix of the standardized device coordinate system and the camera coordinate system
  • the camera extrinsic parameter matrix is the transformation matrix of the world coordinate system and the camera coordinate system.
  • the method includes:
  • the display position of the virtual object in the augmented reality image is controlled to move.
  • the method includes:
  • the virtual object in the augmented reality image is controlled to move to the target display position.
  • an embodiment of the present disclosure further provides an electronic device 1100 .
  • the schematic structural diagram of the electronic device 1100 provided by the embodiment of the present disclosure includes:
  • Embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium.
  • a computer program is stored on the computer-readable storage medium.
  • the steps of the method for controlling a virtual object described in the foregoing method embodiments are executed.
  • the storage medium may be a volatile or non-volatile computer-readable storage medium.
  • the computer program product of the method for controlling virtual objects provided by the embodiments of the present disclosure includes a computer-readable storage medium storing program codes, and the instructions included in the program codes can be configured to execute the virtual objects described in the foregoing method embodiments.
  • the steps of the control method reference may be made to the foregoing method embodiments, which will not be repeated here.
  • Embodiments of the present disclosure also provide a computer program, which implements any one of the methods in the foregoing embodiments when the computer program is executed by a processor.
  • the computer program product can be specifically implemented by hardware, software or a combination thereof.
  • the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.
  • the units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium.
  • the computer software products are stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .
  • the embodiments of the present disclosure disclose a control method, device, electronic device and storage medium for a virtual object, wherein the control method includes: acquiring an augmented reality image in which a real scene image and a virtual object are superimposed; The first display position information of the target key point of the hand in the device coordinate system, and the length information of the arm in the real scene image is identified; based on the first display position information and the length information of the arm, control the The display position of the virtual object in the augmented reality image.
  • the above solution can improve the recognition efficiency of the target key points of the hand, improve the processing efficiency in the process of controlling the virtual object, and optimize the control effect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente divulgation concerne un procédé et un appareil de commande d'objet virtuel, ainsi qu'un dispositif électronique et un support de stockage. Le procédé de commande comprend les étapes consistant à : acquérir une image de réalité augmentée formée par la superposition d'une image de scène de réalité et d'un objet virtuel ; identifier des premières informations de position d'affichage d'un point clé cible d'une main dans l'image de scène de réalité dans un système de coordonnées de dispositif et identifier des informations de longueur d'un bras dans l'image de scène de réalité ; et commander une position d'affichage de l'objet virtuel dans l'image de réalité augmentée sur la base des premières informations de position d'affichage et des informations de longueur du bras.
PCT/CN2021/091072 2020-07-30 2021-04-29 Procédé et appareil de commande d'objet virtuel, dispositif électronique et support de stockage WO2022021980A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010753268.8A CN111880657B (zh) 2020-07-30 2020-07-30 一种虚拟对象的控制方法、装置、电子设备及存储介质
CN202010753268.8 2020-07-30

Publications (1)

Publication Number Publication Date
WO2022021980A1 true WO2022021980A1 (fr) 2022-02-03

Family

ID=73205754

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/091072 WO2022021980A1 (fr) 2020-07-30 2021-04-29 Procédé et appareil de commande d'objet virtuel, dispositif électronique et support de stockage

Country Status (3)

Country Link
CN (1) CN111880657B (fr)
TW (1) TW202205059A (fr)
WO (1) WO2022021980A1 (fr)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114742977A (zh) * 2022-03-30 2022-07-12 青岛虚拟现实研究院有限公司 一种基于ar技术的视频透视方法
CN114911384A (zh) * 2022-05-07 2022-08-16 青岛海信智慧生活科技股份有限公司 镜子显示器及其远程控制方法
CN115861581A (zh) * 2023-02-08 2023-03-28 成都艺馨达科技有限公司 一种基于混合现实的移动互联云服务方法及系统
CN115937430A (zh) * 2022-12-21 2023-04-07 北京百度网讯科技有限公司 用于展示虚拟对象的方法、装置、设备及介质
CN116309850A (zh) * 2023-05-17 2023-06-23 中数元宇数字科技(上海)有限公司 一种虚拟触控识别方法、设备及存储介质
CN116363331A (zh) * 2023-04-03 2023-06-30 北京百度网讯科技有限公司 图像生成方法、装置、设备以及存储介质
CN117032617A (zh) * 2023-10-07 2023-11-10 启迪数字科技(深圳)有限公司 基于多屏幕的网格拾取方法、装置、设备及介质

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111880657B (zh) * 2020-07-30 2023-04-11 北京市商汤科技开发有限公司 一种虚拟对象的控制方法、装置、电子设备及存储介质
CN114584681A (zh) * 2020-11-30 2022-06-03 北京市商汤科技开发有限公司 目标对象的运动展示方法、装置、电子设备及存储介质
CN114584680A (zh) * 2020-11-30 2022-06-03 北京市商汤科技开发有限公司 一种运动数据展示方法、装置、计算机设备及存储介质
CN114584679A (zh) * 2020-11-30 2022-06-03 北京市商汤科技开发有限公司 赛况数据呈现方法、装置、计算机设备及可读存储介质
CN112714337A (zh) * 2020-12-22 2021-04-27 北京百度网讯科技有限公司 视频处理方法、装置、电子设备和存储介质
CN112799507B (zh) * 2021-01-15 2022-01-04 北京航空航天大学 人体虚拟模型展示方法、装置、电子设备及存储介质
CN112817447B (zh) * 2021-01-25 2024-05-07 暗物智能科技(广州)有限公司 一种ar内容显示方法及系统
CN113253838A (zh) * 2021-04-01 2021-08-13 作业帮教育科技（北京）有限公司 基于ar的视频教学方法、电子设备
CN113359985A (zh) * 2021-06-03 2021-09-07 北京市商汤科技开发有限公司 数据展示方法、装置、计算机设备以及存储介质
CN114115528B (zh) * 2021-11-02 2024-01-19 深圳市雷鸟网络传媒有限公司 虚拟对象控制方法、装置、计算机设备和存储介质
CN114422644A (zh) * 2022-01-25 2022-04-29 Oppo广东移动通信有限公司 设备控制方法、装置、用户设备和计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104615242A (zh) * 2009-08-12 2015-05-13 岛根县 图像识别装置、操作判断方法以及程序
CN107077169A (zh) * 2014-11-14 2017-08-18 高通股份有限公司 扩增现实中的空间交互
CN110476142A (zh) * 2017-04-03 2019-11-19 微软技术许可有限责任公司 虚拟对象用户界面显示
CN110941337A (zh) * 2019-11-25 2020-03-31 深圳传音控股股份有限公司 虚拟形象的控制方法、终端设备及计算机可读存储介质
CN111880657A (zh) * 2020-07-30 2020-11-03 北京市商汤科技开发有限公司 一种虚拟对象的控制方法、装置、电子设备及存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9245388B2 (en) * 2013-05-13 2016-01-26 Microsoft Technology Licensing, Llc Interactions of virtual objects with surfaces
US10304248B2 (en) * 2014-06-26 2019-05-28 Korea Advanced Institute Of Science And Technology Apparatus and method for providing augmented reality interaction service
CN108762495B (zh) * 2018-05-18 2021-06-29 深圳大学 基于手臂动作捕捉的虚拟现实驱动方法及虚拟现实系统
CN111103967A (zh) * 2018-10-25 2020-05-05 北京微播视界科技有限公司 虚拟对象的控制方法和装置
CN111161335A (zh) * 2019-12-30 2020-05-15 深圳Tcl数字技术有限公司 虚拟形象的映射方法、映射装置及计算机可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104615242A (zh) * 2009-08-12 2015-05-13 岛根县 图像识别装置、操作判断方法以及程序
CN107077169A (zh) * 2014-11-14 2017-08-18 高通股份有限公司 扩增现实中的空间交互
CN110476142A (zh) * 2017-04-03 2019-11-19 微软技术许可有限责任公司 虚拟对象用户界面显示
CN110941337A (zh) * 2019-11-25 2020-03-31 深圳传音控股股份有限公司 虚拟形象的控制方法、终端设备及计算机可读存储介质
CN111880657A (zh) * 2020-07-30 2020-11-03 北京市商汤科技开发有限公司 一种虚拟对象的控制方法、装置、电子设备及存储介质

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114742977A (zh) * 2022-03-30 2022-07-12 青岛虚拟现实研究院有限公司 一种基于ar技术的视频透视方法
CN114911384A (zh) * 2022-05-07 2022-08-16 青岛海信智慧生活科技股份有限公司 镜子显示器及其远程控制方法
CN114911384B (zh) * 2022-05-07 2023-05-12 青岛海信智慧生活科技股份有限公司 镜子显示器及其远程控制方法
CN115937430A (zh) * 2022-12-21 2023-04-07 北京百度网讯科技有限公司 用于展示虚拟对象的方法、装置、设备及介质
CN115937430B (zh) * 2022-12-21 2023-10-10 北京百度网讯科技有限公司 用于展示虚拟对象的方法、装置、设备及介质
CN115861581A (zh) * 2023-02-08 2023-03-28 成都艺馨达科技有限公司 一种基于混合现实的移动互联云服务方法及系统
CN116363331A (zh) * 2023-04-03 2023-06-30 北京百度网讯科技有限公司 图像生成方法、装置、设备以及存储介质
CN116363331B (zh) * 2023-04-03 2024-02-23 北京百度网讯科技有限公司 图像生成方法、装置、设备以及存储介质
CN116309850A (zh) * 2023-05-17 2023-06-23 中数元宇数字科技(上海)有限公司 一种虚拟触控识别方法、设备及存储介质
CN116309850B (zh) * 2023-05-17 2023-08-08 中数元宇数字科技(上海)有限公司 一种虚拟触控识别方法、设备及存储介质
CN117032617A (zh) * 2023-10-07 2023-11-10 启迪数字科技(深圳)有限公司 基于多屏幕的网格拾取方法、装置、设备及介质
CN117032617B (zh) * 2023-10-07 2024-02-02 启迪数字科技(深圳)有限公司 基于多屏幕的网格拾取方法、装置、设备及介质

Also Published As

Publication number Publication date
TW202205059A (zh) 2022-02-01
CN111880657A (zh) 2020-11-03
CN111880657B (zh) 2023-04-11

Similar Documents

Publication Publication Date Title
WO2022021980A1 (fr) Procédé et appareil de commande d'objet virtuel, dispositif électronique et support de stockage
US9912874B2 (en) Real-time visual effects for a live camera view
US11430192B2 (en) Placement and manipulation of objects in augmented reality environment
CN105612478B (zh) 用户接口程序化的缩放
CN112148197A (zh) 增强现实ar交互方法、装置、电子设备及存储介质
JP7008730B2 (ja) 画像に挿入される画像コンテンツについての影生成
US12100098B2 (en) Simple environment solver using planar extraction
US20130234934A1 (en) Three-Dimensional Collaboration
US20120162384A1 (en) Three-Dimensional Collaboration
WO2021143282A1 (fr) Procédé et appareil de génération de modèle facial tridimensionnel, dispositif informatique et support de stockage
US20140320404A1 (en) Image processing device, image processing method, and program
WO2016122973A1 (fr) Mappage de texture en temps réel
Desai et al. A window to your smartphone: exploring interaction and communication in immersive vr with augmented virtuality
WO2022022141A1 (fr) Procédé et appareil d'affichage d'image, et dispositif informatique et support de stockage
WO2021097600A1 (fr) Procédé et appareil d'interaction inter-air et dispositif
JP2014531693A (ja) 動き制御されたリストスクローリング
US20210073545A1 (en) Object Detection With Instance Detection and General Scene Understanding
Chen et al. A case study of security and privacy threats from augmented reality (ar)
Rocca et al. Head pose estimation by perspective-n-point solution based on 2d markerless face tracking
CN107797648B (zh) 虚拟触摸系统和图像识别定位方法、计算机可读存储介质
WO2019037257A1 (fr) Dispositif et procédé de commande de saisie de mot de passe, et support de stockage lisible par ordinateur
EP3088991B1 (fr) Dispositif portable et procédé permettant une interaction utilisateur
WO2023124691A1 (fr) Affichage de scène de réalité augmentée
TWI740275B (zh) 擴增實境物件顯示裝置及擴增實境物件顯示方法
US11488352B1 (en) Modeling a geographical space for a computer-generated reality experience

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21850663

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21850663

Country of ref document: EP

Kind code of ref document: A1