WO2023201937A1 - 基于故事场景的人机互动方法、装置、设备及介质 - Google Patents

基于故事场景的人机互动方法、装置、设备及介质 Download PDF

Info

Publication number
WO2023201937A1
WO2023201937A1 PCT/CN2022/112406 CN2022112406W WO2023201937A1 WO 2023201937 A1 WO2023201937 A1 WO 2023201937A1 CN 2022112406 W CN2022112406 W CN 2022112406W WO 2023201937 A1 WO2023201937 A1 WO 2023201937A1
Authority
WO
WIPO (PCT)
Prior art keywords
character
story
scene
information
area
Prior art date
Application number
PCT/CN2022/112406
Other languages
English (en)
French (fr)
Inventor
董广泽
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to US18/204,214 priority Critical patent/US20230330541A1/en
Publication of WO2023201937A1 publication Critical patent/WO2023201937A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/25—Output arrangements for video game devices
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50—Controlling the output signals based on the game progress
    • A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/55—Controlling game characters or game objects based on the game progress
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/80—Special adaptations for executing a specific game genre or game mode
    • A63F13/847—Cooperative playing, e.g. requiring coordinated actions from several players to achieve a common goal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment

Definitions

  • Embodiments of the present application relate to the field of augmented reality (Augmented Reality, AR), and in particular to a human-computer interaction method, device, equipment and medium based on story scenes.
  • AR Augmented Reality
  • Script Killing is a game in which multiple players each play a story character, and each story character performs a separate reasoning process.
  • a projector is used to project the projection screen on the wall of the script killing room.
  • This projection screen is used to simulate designated script killing scenes, such as ancient costume wedding scenes, spy scenes of the Republic of China, etc.
  • Players need to change the designated costumes and props, and use paper scripts to complete reasoning in the script killing room.
  • This application provides a human-computer interaction method, device, equipment and medium based on story scenes.
  • the technical solution is as follows:
  • a human-computer interaction method based on story scenes is provided.
  • the method is executed by a terminal with a camera.
  • the method includes:
  • the video picture of the real-scene video stream includes the background area and the foreground character area, and the foreground character area corresponds to the real-scene character;
  • the AR video stream is displayed based on the real-life video stream.
  • the video picture of the AR video stream includes an AR background area and an AR character area.
  • the AR background area displays the scene picture of the story scene.
  • the AR character area displays a person wearing a The real-life character dressed in AR, and the AR dress corresponds to the story character in the story scene;
  • the AR background area is obtained by processing the picture content in the background area
  • the AR character area is obtained by processing the picture content in the foreground character area
  • a human-computer interaction method based on story scenes includes:
  • the picture content in the background area is processed to obtain the AR background area, and the picture content in the foreground character area is processed to obtain the AR character area;
  • the AR background area displays the scene picture of the story scene, and the AR character area displays real-life characters wearing AR costumes.
  • the AR costume corresponds to the story character in the story scene;
  • an AR video stream is obtained
  • a human-computer interaction device based on story scenes includes:
  • the acquisition module is used to obtain the real-scene video stream collected by the camera.
  • the video picture of the real-scene video stream includes a background area and a foreground character area, and the foreground character area corresponds to the real-scene character;
  • a display module is configured to display an AR video stream based on the real-life video stream.
  • the video picture of the AR video stream includes an AR background area and an AR character area.
  • the AR background area displays the scene picture of the story scene.
  • the AR The character area displays the real-life character wearing the AR costume, and the AR costume corresponds to the story character in the story scene;
  • An interactive module configured to change the display content of the AR video stream in response to interactive operations; and complete the reasoning task corresponding to the story scene based on the changed display content;
  • the AR background area is obtained by processing the picture content in the background area
  • the AR character area is obtained by processing the picture content in the foreground character area
  • a human-computer interaction device based on a story scene is provided.
  • the device is connected with a camera, and the device includes:
  • the receiving module is used to receive the real-scene video stream collected by the camera.
  • the video picture of the real-scene video stream includes a background area and a foreground character area, and the foreground character area corresponds to the real-scene character;
  • the processing module is used to process the live video stream into an augmented reality AR video stream.
  • the video picture of the AR video stream includes an AR background area and an AR character area.
  • the AR background area displays the scene picture of the story scene, and the AR character area displays the AR costume.
  • the real-life characters and AR costumes correspond to the story characters in the story scenes;
  • the interactive module is used to complete reasoning tasks corresponding to story scenes based on AR video streams;
  • the AR background area is obtained by processing the picture content in the background area
  • the AR character area is obtained by processing the picture content in the foreground character area.
  • a terminal includes: a processor and a memory.
  • the memory stores a computer program.
  • the computer program is loaded and executed by the processor to implement the above human-computer interaction method based on story scenarios.
  • a computer-readable storage medium stores a computer program.
  • the computer program is loaded and executed by a processor to implement the above human-computer interaction method based on story scenes.
  • a computer program product stores a computer program.
  • the computer program is loaded and executed by a processor to implement the above human-computer interaction method based on story scenarios.
  • the escape story scene can provide a very immersive visual effect without the need for specially customized costumes, props and paper scripts. While reducing the consumption of specially customized physical resources, a better visual experience than related technologies can be achieved.
  • Figure 1 shows a schematic diagram of a real-life video stream collected by a user through a camera provided by an exemplary embodiment of the present application
  • Figure 2 shows a schematic diagram of an AR video stream displayed by a user's AR terminal provided by an exemplary embodiment of the present application
  • Figure 3 shows a structural block diagram of a computer system provided by an exemplary embodiment of the present application
  • Figure 4 shows a flow chart of a human-computer interaction method based on story scenarios provided by an exemplary embodiment of the present application
  • Figure 5 shows a flow chart of a method for displaying an AR video stream based on a real-life video stream provided by an exemplary embodiment of the present application
  • Figure 6 shows a schematic diagram of not displaying real people wearing AR glasses in the AR video stream provided by an exemplary embodiment of the present application
  • Figure 7 is a flow chart of a human-computer interaction method based on story scenes provided by another exemplary embodiment of the present application.
  • Figure 8 shows a flow chart of a method for obtaining evidence collection information of a first story character based on a story scene provided by an exemplary embodiment of the present application
  • Figure 9 shows a flow chart of a method for obtaining evidence collection information of a first story character based on a story scene provided by another exemplary embodiment of the present application.
  • Figure 10 shows a flow chart of a method for obtaining evidence collection information of a first story character based on a story scene provided by another exemplary embodiment of the present application
  • Figure 11 shows a flow chart of a human-computer interaction method based on story scenarios provided by an exemplary embodiment of the present application
  • Figure 12 shows a schematic diagram of dynamic video semantic segmentation provided by an exemplary embodiment of the present application
  • Figure 13 shows a schematic diagram of the FCN network structure provided by an exemplary embodiment of the present application.
  • Figure 14 shows a schematic diagram illustrating the comparison between the FCN semantic segmentation results provided by an exemplary embodiment of the present application and the effect of real samples
  • Figure 15 shows a schematic diagram of an AR screen displayed by a user's AR terminal during the evidence collection stage provided by an exemplary embodiment of the present application
  • Figure 16 shows a game scene flow chart of the human-computer interaction method based on story scenes provided by an exemplary embodiment of the present application
  • Figure 17 shows a structural block diagram of a computer system provided by another exemplary embodiment of the present application.
  • Figure 18 shows a block diagram of a device for human-computer interaction based on story scenes provided by an exemplary embodiment of the present application
  • Figure 19 shows a block diagram of a device for human-computer interaction based on story scenes provided by another exemplary embodiment of the present application.
  • Figure 20 shows a structural block diagram of a terminal provided by an exemplary embodiment of the present application.
  • Figure 21 shows a structural block diagram of a server provided by an exemplary embodiment of the present application.
  • Reasoning mission Also known as a reasoning game, it is a task for one or more players to solve puzzles based on clues in the story scene.
  • Traditional story scenes mainly rely on text on paper media to create.
  • script-killing games and escape room games that have become popular offline in recent years, the story scenes are created by game venues built by merchants.
  • Script Killing Originated from a type of live-action role-playing game.
  • the prototype is called Mystery of Murder.
  • the game is based on the script.
  • the host guides the progress of the game.
  • Players collect evidence through multiple rounds. , speaking, and reasoning links to complete the reasoning tasks of your own story characters, and restore the process of the incident (the modus operandi).
  • the story character needs to decipher the method used by the murderer to kill people in a secret room.
  • the relationships between the story characters in the script are also intricately related.
  • Players need to immerse themselves in the story characters and carefully consider the speeches and information of the players present, and finally vote for the murderer they identify.
  • the host will reveal the truth and Conduct a game review.
  • Some scripts will also trigger one of multiple endings based on the player's choice. Such scripts with multiple endings are called "mechanical scripts”.
  • Escape room a real-life escape game.
  • the main ideas of the game are mostly derived from scenes such as film and television dramas, books and the Internet.
  • players usually play the protagonist from a first perspective or a third perspective, and are limited to an almost completely closed or threatening environment.
  • Within the environment i.e., "secret room”
  • secret room there is at least one secret room in a single game.
  • Story scene Each reasoning task corresponds to a story.
  • the time, place and environment of the story constitute the story scene of the reasoning task.
  • the spy scenes of the Republic of China the scenes of immortals cultivating immortals, the scenes of western cowboys and the scenes of exploring ancient tombs, etc.
  • Each reasoning task has at least one character. Different characters have different genders, different images, different personalities, story backgrounds, plot promotion functions, and reasoning tasks.
  • the story character can be a virtual character, a virtual animal, an animation character, etc.
  • Collect evidence (referred to as evidence search): Each reasoning task has at least one piece of evidence. Different evidence has different plot-promoting effects and is obtained by players after performing corresponding operations on virtual props or other story characters.
  • Role information Different characters have different role information, such as name, age, gender, appearance, personality, growth background, social relationships, schedule, etc.
  • Public information It is character information or evidence collection information that all story characters (or at least two story characters) in the reasoning task have permission to view.
  • Private information It is character information that only a certain story character in the reasoning task has permission to view. For example, among the multiple pieces of information about the character in the first story, there is information A that only the character in the second story has permission to view. Then information A is the private information of the character in the second story.
  • This application provides an interactive solution based on AR that presents reasoning tasks of story scenes to users.
  • the reasoning task can be a game task with reasoning elements such as script killing and escape room.
  • This application can present at least one of story scenes, story characters, and character information to players using AR technology.
  • the AR terminal can be at least one of a smartphone, a tablet computer, an e-book reader, a laptop computer, a desktop computer and AR glasses.
  • User A, User B, User C and User D sit around a four-person table in a real scene, and User E stands next to the four-person table.
  • user A, user B, user C, user D and user E all hold an AR terminal.
  • One of the five users selects a reasoning task among at least two candidate story scenarios.
  • the AR terminal After the AR terminal performs image semantic recognition on the real scene image, it obtains the background area 101 and the foreground person area 102.
  • the foreground person area 102 includes a face area 1021 and a non-face area 1022.
  • facial recognition is performed on the face area 1021, the identity of the real person is obtained and bound to the story character selected by the player. For example, user A is bound to the first story role, user B is bound to the second story role, user C is bound to the third story role, user D is bound to the fourth story role, and user E is bound to the fifth story role.
  • Figure 1 shows a schematic diagram of a real scene picture 100 collected by user E's camera.
  • the real scene picture includes a background area 101 and a foreground character area 102.
  • the foreground character area 102 corresponds to the real scene characters User A, User B, User C and User D. .
  • the background area 101 is shown as the interior of a room with low cabinet furnishings.
  • the real person area includes a face area 1021 and a non-face area 1022.
  • User A's camera faces user E.
  • the face area 1021 shows his true appearance and glasses, and the non-face area 1022 shows that he has a bun and is wearing a sleeveless top.
  • User B faces the camera of user E.
  • the face area 1021 shows his true appearance, and the non-face area 1022 shows that he has figure bangs and shoulder-length hair, and is wearing a V-neck half-sleeved top.
  • User C faces the camera of user E.
  • the face area 1021 shows his true appearance, and the non-face area 1022 shows that he has shoulder-length hair and is wearing a camisole.
  • User D faces away from user E's camera, and the non-face area 1021 shows that his hair is slightly curly and he is wearing a short-sleeved top.
  • Each user's AR terminal will replace the background area 101 in the real scene picture 100 with the AR background area 201 based on the scene material of the story scene, and replace the non-face area 1022 of the real scene character based on the character material of the story character, thereby replacing the foreground character.
  • the area 102 is replaced with the AR character area 202, thereby replacing the real scene picture 100 with the AR picture 200.
  • the AR terminal will also display the AR information control 203 in the AR screen.
  • the player can obtain character information corresponding to other story characters based on the AR screen 200 .
  • the character information acquisition methods include: obtaining from the server, voice input (Optical Character Recognition, OCR) scanning and at least one of keyboard input.
  • voice input Optical Character Recognition, OCR
  • OCR Optical Character Recognition
  • players can operate or interact with at least one of the story characters, AR props and AR scenes, and the story characters corresponding to themselves will also perform the same actions.
  • the evidence collection information of other story characters in the story scene is obtained, and the information is displayed in the AR information control 203.
  • user D obtains the basic role information of the bound fourth story character from the server: character four, female, 16 years old, daughter of Imperial Physician Yang; in the public chat stage, user A and user B , User C and User E learned that Character Four had always been on good terms with the victim, but recently the relationship had deteriorated; User E chatted privately with Character Four and learned that Character Four had been to the scene of the crime yesterday; User E searched Character Four’s box for evidence Later, a damaged silver hairpin was found.
  • Figure 2 shows an AR screen 200 displayed by the AR terminal of user E.
  • the screen includes an AR background area 201, an AR character area 202, and an AR information control 203.
  • the AR background area 201 displays the story scene of the reasoning task, and the AR character area 202 A real person wearing an AR costume 204 is displayed.
  • the AR costume 204 corresponds to the story character in the story scene.
  • the AR information control 203 displays the evidence collection information of the story character (at least one of basic information, public information and private information).
  • the AR information The control 203 may be located around a story character or at a location where evidence information is obtained.
  • the AR background area 201 is displayed under a willow tree at the foot of a mountain outside the city.
  • the AR character area 112 displays user A, user B, user C and user D wearing different antique AR costumes, and user B is playing the AR virtual guqin.
  • the information of the fourth story character bound by user D is displayed on the AR information control 120 on its side.
  • the information obtained by User E in the private chat stage is private information and cannot be viewed by User A, User B, User C, and User D.
  • the information obtained by User E in the evidence collection stage chooses to be made public to other players.
  • User A, User B, and User C , user D can all view.
  • FIG. 3 shows a structural block diagram of a computer system provided by an exemplary embodiment of the present application.
  • the computer system 300 includes a first terminal 310, a server 320 and a second terminal 330.
  • the first terminal 310 has a camera, and is installed and runs an application supporting AR interaction and reasoning tasks.
  • the first terminal 310 is an AR terminal used by the first user.
  • the first terminal 310 is connected to the server 320 through a wireless network or a wired network.
  • the server 320 includes one of a server, multiple servers, a cloud computing platform, and a virtualization center.
  • the server 320 includes a processor 321 and a memory 322.
  • the memory 322 further includes a receiving module 3221, a display module 3222 and a control module 3223.
  • the server 320 is used to provide background services for applications that support AR interaction and reasoning tasks.
  • the server 320 is responsible for the main computing work, and the first terminal 310 and the second terminal 330 are responsible for the minor computing work; or the server 320 is responsible for the minor computing work, and the first terminal 310 and the second terminal 330 are responsible for the major computing work. ;
  • the server 320, the first terminal 310 and the second terminal 330 use a distributed computing architecture to perform collaborative computing.
  • the second terminal 330 has a camera, and is installed and runs an application supporting AR interaction and reasoning tasks.
  • the second terminal 330 is an AR terminal used by the second user.
  • first story character and the second story character are in the same story scene.
  • first story character and the second story character may belong to the same team, the same organization, have a friend relationship, or have temporary communication permissions.
  • the application programs installed on the first terminal 310 and the second terminal 330 are the same, or the application programs installed on the two terminals are the same type of application programs on different control system platforms.
  • the first terminal 310 may generally refer to one of multiple terminals, and the second terminal 330 may generally refer to one of multiple terminals.
  • This embodiment only takes the first terminal 310 and the second terminal 330 as an example.
  • the device types of the first terminal 310 and the second terminal 330 are the same or different, and the device types include: at least one of a smart phone, a tablet computer, an e-book reader, a laptop computer, a desktop computer, and AR glasses.
  • the following embodiment illustrates that the terminal includes a mobile phone and AR glasses.
  • terminals or story characters may be more or less. For example, there may be only one terminal or story character, or there may be dozens, hundreds, or more terminals or story characters.
  • the embodiments of this application do not limit the number and device types of terminals or story characters.
  • the information including but not limited to user equipment information, user personal information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • signals involved in this application All are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
  • the live video streams involved in this application are all obtained with full authorization.
  • the terminals and servers only cache the live video streams during the running of the program and will not solidify, store and reuse the live video streams. related data.
  • Figure 4 shows a flow chart of a human-computer interaction method based on story scenarios provided by an exemplary embodiment of the present application.
  • the method is executed by the terminal shown in FIG. 3 as an example.
  • the terminal has a camera.
  • the method includes:
  • Step 220 Obtain the real-scene video stream collected by the camera.
  • the video picture of the real-scene video stream includes the background area and the foreground character area;
  • An application supporting AR interaction and reasoning tasks is installed and run in the terminal, and the reasoning tasks may be at least one of reasoning games such as script killing or escape room.
  • the terminal After the terminal receives the user's operation to start the application, it displays at least two candidate story scenes. After the terminal receives the user's operation to select any story scene, it displays at least two candidate story characters. The terminal receives After the user selects any story character, the user's facial data is also bound to the selected story character.
  • the terminal After the terminal receives the user's operation to start the application, it displays at least two candidate story scenes. After the terminal receives the user's operation to select any story scene, it displays at least two candidate story characters. The terminal receives After the user selects any story character, the user's facial data is also bound to the selected story character.
  • the terminal obtains the live video stream collected by the camera.
  • the real-view video stream includes multiple real-view video frames, and each real-view video frame constitutes a real-view video picture. Multiple real-life video frames are displayed in a time sequence as a continuous video picture.
  • the real-scene video frame is divided into a background area and a foreground character area through image semantic recognition, and multiple real-scene video frames are arranged in time sequence and displayed as a real-scene video stream. All or part of the live video frames of the live video stream include a background area and a foreground character area.
  • the background area refers to the scenery or scenery area that sets off the real-life characters in the real-life video frames collected by the terminal's camera.
  • the background area is the walls and furniture of the room.
  • the foreground person area refers to the real person area collected by the terminal's camera.
  • Real-life character areas include face areas and non-face areas.
  • the face area refers to the face area of the real person in the real scene captured by the camera.
  • the non-face area refers to the area other than the face area of the real person in the real scene captured by the camera, such as the torso area and limbs area.
  • Image semantic recognition refers to the technology of computer processing, analysis and understanding of images to identify the two-dimensional areas where different semantic objects are located in the same video frame or image. For example, in the same live video frame, the background area, face area and non-face area are distinguished.
  • Step 240 Display the AR video stream based on the real-life video stream
  • the AR video stream includes multiple AR video frames.
  • Each AR video frame constitutes an AR video picture.
  • the multiple AR video frames are arranged in time sequence and displayed as a continuous AR video picture.
  • the AR video frame includes an AR background area and an AR character area.
  • the AR character area displays real-life characters wearing AR costumes, and the AR costumes correspond to story characters in the story scene.
  • the AR background area is obtained by processing the picture content in the background area, and the AR character area is obtained by processing the picture content in the foreground character area.
  • AR background area refers to the virtual background displayed when the application is running.
  • the AR background area replaces the background area in the real video frame based on the scene material of the story scene, and integrates the virtual content and the real content in real time to form A three-dimensional environment for interaction between virtual and reality.
  • AR character area refers to the area of real-life characters wearing AR costumes displayed when the application is running.
  • the AR character area replaces the non-face areas of the real-life characters based on the character materials of the story characters, and combines virtual content and real-life characters.
  • the facial area is fused in real time to form a three-dimensional character area that interacts between virtual and reality.
  • Each story character has at least one set of AR costumes.
  • a story character has different AR costumes in different story scenes, a story character has different AR costumes in different time periods of the same story scene, or a story character has different AR costumes in the same story scene.
  • Step 260 In response to the interactive operation, change the display content of the AR video stream.
  • display content of the AR background area is changed in response to item interaction with virtual items in the AR background area.
  • the object interaction operation includes at least one of an object touching operation, an object grabbing operation, an object using operation, an object checking operation, a gesture pointing operation, an eyeball locking operation, and an eyeball sliding operation.
  • Item touch operation refers to the operation of touching virtual items.
  • the item grabbing operation refers to the operation of grabbing virtual items.
  • Item usage operation refers to the operation of using virtual items.
  • the item checking operation refers to the operation of checking virtual items.
  • the method of changing the display content of the AR background area includes one of the following:
  • the way to escape the escape room is displayed in text in the AR background area.
  • the secret room scene displayed in the AR background area is updated to an outdoor scene.
  • character information of the real-life character is changed in response to character interaction with the real-life character in the AR character area.
  • the character interaction operation includes at least one of a character touch operation, a character grasping operation, a character talking operation, a gesture pointing operation, an eyeball locking operation, and an eyeball sliding operation.
  • the role information includes at least one of gender, age, identity, occupation, expertise, ability, skills, height, weight, and clothing.
  • changing the role information of the real-life character includes one of the following:
  • the role information includes occupation. After talking with User B, User A learns that User B's occupation is "Doctor” rather than “Nurse”, so User B's occupation is changed from “Nurse” to "Doctor” .
  • story character A asks story character B a question
  • story character B answers story character B's occupation
  • story character A needs to fight story character B. If story character A defeats story character B, the character information of story character B is deleted.
  • story character A after story character A talks with story character B, story character A learns that story character B and story character C are brothers, and adds new role relationship information between story character B and story character C. .
  • character interaction operations in the above different embodiments may be the same operation or may be different operations.
  • the scene picture of the story scene is changed.
  • the scene switching operation includes at least one of an object touching operation, an object grabbing operation, a character touching operation, a character grasping operation, a gesture pointing operation, an eyeball locking operation, and an eyeball sliding operation.
  • the first scene picture of the story scene is switched to the second scene picture, and the first scene picture and the second scene picture are different scene pictures.
  • storyline-related display content in the AR video stream is changed in response to a storyline triggering operation of the story scene.
  • the storyline triggering operation includes at least one of the following: the storyline develops to a preset time point, an operation triggered by a story character to advance the development of the storyline, and an operation triggered by the host to advance the development of the storyline.
  • the method of changing the display content related to the storyline in the AR video stream includes one of the following:
  • the scene switching operation includes at least one of an object touching operation, an object grabbing operation, a character touching operation, a character grasping operation, a gesture pointing operation, an eyeball locking operation, and an eyeball sliding operation.
  • the story scene is switched from an ancient fairy tale scene to a modern urban scene.
  • a newly added plot prop is displayed in the AR background area, and the plot prop is used to promote the development of the storyline.
  • story character A tells story character A's profession to story character B
  • story character A's profession is updated.
  • storyline triggering operations in the above different embodiments may be the same operation or different operations.
  • Step 280 Based on the changed display content, complete the reasoning task corresponding to the story scene.
  • the AR information includes AR information associated with the AR character displayed on the sides, AR information associated with the virtual props displayed on the sides, AR information associated with the virtual environment, AR information associated with the Non-Player Character (NPC). ) at least one of the AR information displayed on.
  • NPC Non-Player Character
  • the AR information includes at least one of text information, picture information, video information, audio information, animation information, and special effects information related to completing the reasoning task.
  • Information acquisition tasks are tasks used to obtain character information for each story character.
  • Evidence collection tasks are tasks used to collect relevant evidence information for reasoning tasks.
  • the puzzle reasoning task is a task of performing puzzle reasoning based on the acquired character information and/or relevant evidence information.
  • the reasoning task includes at least one of: an information acquisition task, an evidence collection task, and a puzzle reasoning task.
  • the information acquisition task includes at least one of an introduction stage, a public chat stage, a private chat stage, an evidence collection stage, and a case closing stage.
  • the evidence collection task includes the evidence collection stage, and the puzzle reasoning task includes the case closing stage.
  • the player obtains the background information of the story during the script introduction stage, obtains basic information about the story characters during the public chat stage, obtains the private extended information about the story characters during the private chat stage, and obtains information about virtual scenes, virtual props or virtual scenes during the evidence collection stage.
  • the NPC performs evidence collection operations and obtains evidence collection information. After analysis and voting, the reasoning results are obtained.
  • the terminal displays or the host announces whether the results are correct or not to complete the reasoning task.
  • the reasoning task includes at least one of: an information acquisition task, an evidence collection task, and a puzzle reasoning task.
  • the information acquisition task includes the information seeking stage
  • the evidence collection task includes the evidence collection stage
  • the puzzle reasoning task includes the escape stage.
  • players can know the story background, reasoning tasks or escape goals of this escape room through the terminal, server or staff.
  • the player learns the story background and escape goal of the escape room by reading the virtual information control displayed on the terminal, and obtains the method of escaping the escape room after conducting evidence collection operations on the virtual scene, virtual props or NPCs in the evidence collection stage. After successfully escaping the secret room, complete the reasoning task.
  • the interactive reasoning task is a task of interacting with the scene pictures of the story scene, or the interactive reasoning task is a task of interacting with the story characters in the story scene.
  • the player learns by reading the virtual information control displayed on the terminal that the way to escape from the secret room is to complete the puzzle in the story scene.
  • the player puts the puzzle pieces into the designated position to complete the puzzle and meet the conditions for escaping from the secret room. After completing the puzzle, complete the interactive reasoning task.
  • the player learns that the task to enter the next stage of the script killing is to obtain the key held by the story character. After interacting with the story character in the evidence collection stage, the player obtains the key held by the story character and proceeds to the next stage of the script killing. stage, after obtaining the key, complete the interactive reasoning task.
  • the method provided by this embodiment achieves this by replacing the background area in the live video stream with the AR background area corresponding to the reasoning task, and replacing the real people in the live video stream with real people wearing AR costumes.
  • Using AR scenes and AR costumes to create story scenes of script killing or escape rooms can provide very immersive visual effects without the need for specially customized costumes, props and paper scripts. While reducing the consumption of specially customized physical resources, a better visual experience than related technologies can be achieved.
  • step 240 includes step 241, step 243, step 245, step 247 and step 249.
  • Figure 5 shows a flowchart of a method for displaying an AR video stream based on a real-life video stream provided by an exemplary embodiment of the present application. This embodiment illustrates that the method is executed by the terminal and/or server shown in FIG. 3 .
  • the method includes:
  • Step 241 Perform image semantic recognition on the video frames in the live video stream to obtain the background area and the foreground character area.
  • the foreground character area includes the face area and the non-face area;
  • User A, User B, User C and User D are sitting around a four-person table in a real scene.
  • User E is standing next to the four-person table.
  • the cameras of the AR terminals used by each user capture different real scenes.
  • Video stream, the real-scene video stream is uploaded to the server by each terminal respectively.
  • the real-scene video stream contains multiple real-scene video frames.
  • the server performs image semantic recognition on each real-scene video frame to obtain the background area and foreground in each real-scene video frame. Character area.
  • the server can identify the live video stream as shown in Figure 1 after analysis.
  • the background area shows the interior of the room and the low cabinet
  • the foreground character area shows user A, user B, and user The actual face, torso, and limbs of C and user D.
  • Step 243 Perform facial recognition on the face area to obtain the identity of the real person
  • the facial area of each real-life figure recognized by the server corresponds to a set of facial data. After the server identifies the facial data, it can determine the identity of each real-life figure.
  • the facial areas of User A, User B, User C, User D and User E each correspond to a set of face data in the server.
  • the camera of the AR terminal held by User E is suitable for the four users.
  • anyone who collects the face can get their corresponding real-life identity.
  • Step 245 Based on the character identity of the real-life character, determine the first story role bound to the real-life character in the reasoning task;
  • User A, User B, User C, User D and User E each use an AR terminal to select a story character, and the server binds their facial data to the story character. .
  • one of the five users selects or assigns a story role, and the server assigns each user's Facial data is bound to its corresponding story character.
  • the server stores the binding relationship between each user's facial data and story characters. After the inference task is started, after the identity of the real person appearing in the real video frame is recognized, the first story role bound to the real person in the inference task is determined based on the binding relationship.
  • Step 247 Replace the background area based on the scene material of the story scene to obtain the AR background area; replace the non-face area based on the character material of the first story character to obtain the AR character area and obtain the AR video stream;
  • the server determines scene materials for story scenes and character materials corresponding to each story character.
  • the server calculates the content in the background area of the live video streams collected by User A, User B, User C, User D and User E respectively, and renders the scene materials of the story scene to the background area.
  • the server calculates the content in the non-face area of the foreground character area corresponding to each user, renders the character material of the bound story character to the foreground character area, and obtains a real scene wearing AR clothing Character, get the AR character area.
  • the server renders the scene materials of the story scene into the background area of the real-life video stream collected by user A, user B, user C, user D and user E.
  • the server binds user A to The character material of the determined first story character is rendered to the non-face area of user A collected by other users to obtain an AR video stream.
  • the server calculates the content in the background area in the real-life video stream collected by user E, renders the scene materials of the story scene to the background area, and obtains the AR background area; the server 320 calculates the content corresponding to each user.
  • the content in the non-face area of the foreground character area is calculated, and the character material of the bound story character is rendered to the foreground character area to obtain a real-life character wearing an AR costume, and the AR character area is obtained.
  • the server renders the scene materials of the story scene into the real-life video stream collected by user E, and renders the character materials of the story characters bound by each user to the non-face area collected by user E, and obtains AR video streaming.
  • the AR costume of the story character can be created locally and uploaded to the server, or customized locally and uploaded to the server.
  • the AR costume of the story character can be created locally, such as modeling, etc., and the created AR costume can be uploaded to the server; or, the AR costume can be customized, such as adjusting the existing AR costume, such as resizing. , shape, spacing, color change, etc., upload the customized AR decoration to the server.
  • Step 249 Display the AR video stream based on the AR background area and the AR character area.
  • the AR video stream displayed by each user's terminal is collected by the camera of the respective terminal, and then uploaded to the server for processing, and then transmitted back to the terminal by the server.
  • the terminal he uses uploads a live video stream, and after processing by the server, the AR background area and AR character area are obtained.
  • the AR character area corresponds to user B, user C, user D and user E wearing AR costumes.
  • the AR video stream displayed by each user's terminal is collected by the camera of any one of the terminals and uploaded to the server for processing, and then transmitted back to all terminals by the server.
  • the terminal used by user E uploads a live video stream, and after processing, the AR background area and AR character area are obtained.
  • the AR character area corresponds to user A, user B, user C, user D and user E wearing AR costumes.
  • Each terminal selectively displays the AR video stream from its corresponding perspective.
  • the above process can be executed multiple times by the computer system in one reasoning task game.
  • the above processes can also be executed by the terminal without the cooperation of the server.
  • the method provided by this embodiment uses image semantic segmentation to identify the background area and the foreground character area in the live video stream.
  • the foreground character area includes the face area and the non-face area.
  • the AR terminal is AR glasses.
  • the real characters displayed in the live video stream are wearing AR glasses. Since the AR glasses may not match the story scene, for example, when the story scene is an ancient fairy tale scene, the AR glasses, which are modern consumer electronic devices, do not match the ancient visual scene. , therefore, the embodiment of the present application provides a method of not displaying AR glasses worn by real-life characters in the AR video stream in the human-computer interaction method based on story scenes.
  • Figure 6 shows a schematic diagram of a method for not displaying real people wearing AR glasses in the AR video stream provided by the embodiment of the present application.
  • the server inputs the sample face data of the real person collected and uploaded by the terminal and the first facial image 601 displayed in the AR video stream to the generative network for image reconstruction, and obtains the second image of the real person without wearing the AR terminal.
  • the face image 603 is displayed in the second face area of the AR character area in the AR video stream.
  • the generative network is a neural network with image reconstruction capabilities.
  • the generative network includes a discriminator and a generator. During the training process, the discriminator and the generator need to be trained together; during the application process, only the generator is needed.
  • the training set includes multiple sets of sample data of different users.
  • Each set of data includes sample facial images of the same user (wearing an AR terminal) and sample face data (not wearing an AR terminal, such as the user’s front face). image).
  • the computer device inputs the sample face image and the sample face data of the same user into the generator, and the generator reconstructs the predicted face image.
  • the sample face image and the predicted face image have the same face angle (which may be any angle), but the predicted face image does not wear an AR terminal.
  • the sample face data is a frontal face image without an AR terminal, which is used to simulate the user's face collected during the binding stage.
  • the face angle of the sample face picture and the face angle of the sample face data may be different.
  • the discriminator is used to distinguish the predicted face image or the sample face image and identify whether it is an image reconstructed by the generator or an original image. Based on the alternating training method of the generative network, the network parameters of the discriminator are fixed and the network parameters of the generator are updated; or the network parameters of the generator are fixed and the network parameters of the discriminator are updated. Until the error converges or the number of training times reaches the preset number, the trained discriminator and generator are obtained.
  • the computer device inputs the sample face data of the real person collected and uploaded by the terminal and the first face picture displayed in the AR video stream to the generator for image reconstruction, and obtains the second face of the real person without wearing the AR terminal. part screen.
  • Figure 7 shows a flow chart of a human-computer interaction method based on story scenes provided by another exemplary embodiment of the present application.
  • this method is executed by the terminal and/or server shown in Figure 3, so that User A, User B, User C, User D and User E each hold a terminal as shown in Figure 1 to perform the same reasoning task.
  • the terminal has a camera.
  • the method includes:
  • Step 211 Display task selection controls for at least two candidate story scenes
  • the task selection control is a control for selecting a story scene among at least two candidate story scenes.
  • Task selection controls can be displayed as drop-down controls, card controls, or check controls.
  • the terminal After receiving the user's start operation of starting the reasoning task, the terminal displays task selection controls for at least two candidate reasoning tasks, and each reasoning task corresponds to a story scene.
  • the user can perform touch operations on the application (including but not limited to clicks, double-clicks, and slides, etc.) to start inference tasks.
  • the application interface displays at least two candidate inference tasks, such as espionage missions and western cowboy tasks. Type tasks, ancient fantasy tasks and ancient tomb exploration tasks.
  • the cover of each reasoning task has a brief introduction to its story scene, and users can view it by sliding the mobile phone interface.
  • the user can perform touch operations (including but not limited to click, double-click, pull, drag, slide, etc.) on the virtual selection controls suspended in the air or flat on the table in front of them to start inference.
  • Tasks at least two candidate reasoning tasks are displayed on the virtual selection control, such as spy missions, western cowboy tasks, ancient fantasy tasks, and tomb exploration tasks.
  • the cover of each reasoning task has a brief introduction to its story scene. , users can view it through sliding operations or flipping operations.
  • Step 212 In response to the selection operation on the task selection control, determine the selected story scene among the at least two candidate story scenes;
  • the selection operation is an operation used to select the story scene displayed in the task selection control.
  • the selection operation can be in the form of sliding to select in the drop-down control, dragging to select in the card control, or clicking to select in the checkbox control.
  • the virtual selection controls are laid out on the table in front of them. Five users can slide to view each reasoning task. User E drags the "Ancient Story 1" selection control to any user. User A The AR glasses of User B, User C, User D and User E all display the story scene of "Ancient Story 1".
  • Step 213 Display character selection controls for at least two candidate story characters in the story scene
  • the character selection control is a control for selecting a story character among at least two candidate story characters.
  • the role selection control can be displayed as a drop-down control, card control or check control.
  • At least five candidate story characters are displayed on the mobile phone interface, for example: character one, character two, character four, character three and character five.
  • the virtual selection control is suspended in the air and displays virtual selection controls for at least 5 candidate story characters, such as: character one, character two, character four, character three and character five.
  • Step 214 In response to the selection operation on the character selection control, determine the selected story character among the at least two candidate story characters;
  • user A drags the virtual selection control of role one to himself to select role one
  • user E drags the selection controls of role two, role three, role four and role five to user B respectively.
  • User C User D and myself to complete the story character selection.
  • Step 215 Bind the selected story character with the facial data of the real-life character corresponding to the terminal;
  • each user uses the camera of the terminal he holds to collect his or her own facial data.
  • the mobile phone uploads the collected facial data to the server, and the server binds the facial data to the story character selected by the user. For example, user A's facial data is bound to character 1, and the user B's facial data is bound to character two, user C's facial data is bound to character three, user D's facial data is bound to character four, and user E's facial data is bound to character five.
  • the mobile phone calculates the collected facial data, and binds the facial data to the story character selected by the user. For example, user A's facial data is bound to character one, user B's The facial data of user C is bound to character two, the facial data of user C is bound to character three, the facial data of user D is bound to character four, and the facial data of user E is bound to character five.
  • the AR glasses held by user E collect facial data of user A, user B, user C and user D.
  • user E's AR glasses upload the facial data of the four users to the server, and the server compares the facial data of user A, user B, user C, and user D with their selected characters, respectively. Role two, role three and role four are bound.
  • the AR glasses held by user A collect the facial data of user E and upload it to the server.
  • the server binds the facial data of user E with the selected role five.
  • user E's AR glasses calculate the facial data of these four users, and compare the facial data of user A, user B, user C, and user D with their selected characters one, two, and three respectively. Character three and character four are bound.
  • the AR glasses held by user A collect the facial data of user E and perform calculations to obtain the facial data of user E.
  • the facial data is bound to the selected character five. .
  • the server will compare the facial data of the person collected by the camera of the terminal held by any user with the facial data of the person in the story scene. Bind NPCs that have no role in promoting the plot. For example, the cleaning staff suddenly enters the room. User E's camera collects the cleaning staff's facial data and uploads it to the server through the AR terminal. The server binds his facial data to the NPC sweeping maid in the story scene.
  • the terminals of User A, User B, User C, User D and User E display that the cleaning staff is a maid wearing an antique AR costume who is sweeping the floor.
  • Step 220 Obtain the real-scene video stream collected by the camera.
  • the real-scene video stream includes a background area and a foreground character area;
  • the camera of the AR terminal he holds captured the real-life video stream as shown in Figure 1.
  • the server identifies the background area and the foreground character area through image semantic recognition.
  • the background area is a room with low cabinets and furnishings
  • the foreground character area is a room showing real life. Looks like User A, User B, User C and User D.
  • the terminal after the camera collects the live video stream, the terminal performs image semantic recognition on the live video stream and identifies the background area and the foreground character area.
  • the background area is a room with low cabinet furnishings
  • the foreground character area is Display the real appearance of User A, User B, User C and User D.
  • Step 240 Display the AR video stream based on the real-life video stream.
  • the AR video stream picture includes an AR background area and an AR character area;
  • each user's terminal replaces the scene material and character material of the story scene into the real-life video stream collected by its camera.
  • the scene material or character material can be obtained by the terminal from the server. , or the scene material or character material is read from the terminal.
  • the server replaces the scene material and character material of the story scene into the real-life video stream uploaded by user E, and then streams the replaced AR video back to the terminals of the five users, each terminal according to its own perspective Display the corresponding AR video stream.
  • the scene material is a three-dimensional scene in which virtual content and real content are integrated in real time based on the time and space arrangement of the real environment where the user is; or the scene material is a three-dimensional scene that is consistent with the real environment where the user is. Three-dimensional scenes inconsistent with time and space.
  • the terminals held by User A, User B, User C, User D and User E can all obtain the scene materials of "Ancient Story One" from the server (such as the foot of the mountain outside the city, Fuju Tower, role one, role two, role 4. The bedrooms of character three and character five, etc.) and character materials (such as the costumes of character one, character two, character four, character three and character five, etc.), and then the scene materials and character materials are replaced by their respective terminals. in the captured live video stream.
  • the five users are all located at the foot of a mountain outside the city.
  • the scene has AR materials such as mountains, water, trees, sky, and terrain that are inconsistent with the three-dimensional structure of the room where the users are actually located, creating a broader Outdoor visual effects.
  • the bedrooms of the five story characters are arranged according to the room where the user actually lives. For example, the virtual wall is rendered to the real wall, the virtual bed is rendered to the real corner, and the virtual table and cabinet are rendered to the real table and cabinet. wait.
  • the reasoning task includes at least one of: information acquisition task, evidence collection task and puzzle reasoning task.
  • Step 262 Obtain the character information of the first story character
  • the terminal may obtain the character information of the first story through at least one of server acquisition, voice input, OCR scanning, and keyboard input.
  • the server stores character information of all story characters. After the reasoning task enters a specific stage, the terminal can automatically obtain the corresponding character information from the server.
  • the user can enter the character information into the terminal through voice.
  • the user holds a paper script, reads the text content on the paper script, and enters the character information of the story characters on the paper script into the terminal through voice.
  • users can use OCR to scan pictures, paper scripts, virtual paper props, etc. containing story character information to obtain corresponding information.
  • the user can input the character information of the story character into the terminal through keyboard input.
  • the keyboard can be displayed on a terminal interface with a camera such as a smartphone, tablet computer, portable computer, e-book reader, etc.
  • the keyboard can also be a virtual keyboard displayed by AR glasses.
  • user E reads the identity information and interpersonal relationship profile of character five obtained from the server.
  • User A and user C obtain the identity information and interpersonal relationship profile of character five through voice recording of user E's voice.
  • User B The identity information and interpersonal relationship profile of character five are obtained by OCR scanning the information pictures on the desktop.
  • User D inputs the identity information and interpersonal relationship profile of character five through the mobile phone keyboard or AR virtual keyboard.
  • user E chats privately with user D and learns that the character four bound to user D went to the crime scene in Fuju Building the day before the crime, and inputs this information into the AR terminal through the keyboard.
  • Step 264 Display the first AR information in the AR video stream.
  • the first AR information is used to associate the character information with the real-life character corresponding to the first story character for display;
  • user B's AR terminal displays a first AR information control located around user A, and the character information about character one obtained by user B is displayed on the first AR information control.
  • Users C, D and E can also view the public information about character 1 obtained by user B during the public chat stage on the first AR information control displayed on the terminal they hold and located on the side of user A; user B can The private information about character one obtained during the private chat phase is not displayed on the first AR information control located on the side of user A displayed on the terminals held by users C, D and E respectively.
  • the first AR information control about character five is displayed around user E, and the information related to character five obtained by other users during the reasoning task will be displayed on the first AR information control.
  • the information that user E obtained during the private chat stage that character four visited the crime scene the day before the crime is displayed on the first AR information control located on the side of user D, and because this information is user E’s Users A, B, C, and D cannot see the information on the first AR information control on the side of user D.
  • the obtained role information can be sorted according to the user's acquisition time or the timeline in the reasoning task, so as to facilitate the user's viewing, analysis, and reasoning.
  • the information in the information control may be displayed in at least one of the following forms: text description, picture description, voice description, or video playback.
  • the method provided by this embodiment provides a richer game experience by displaying the reasoning tasks of at least two story scenes and the story characters of at least two story scenes, and receiving the user's selection operation; and combining the story characters with the story characters. Bind the user's facial data and replace the live video stream with an AR video stream to provide a more immersive visual experience and sense of substitution; by receiving the role information and evidence collection information acquisition operations of the first story character, and in the AR video Character information and evidence collection information are displayed in the stream, which reduces the cost of information recording, makes information viewing more interesting, and also facilitates users to bind story characters and information.
  • Step 266 Obtain the evidence collection information of the first story character
  • the user can perform evidence collection operations on the virtual scene in the story scene.
  • the operations include but are not limited to touching, clicking, zooming in, zooming out, splicing, etc., to obtain evidence collection information in the virtual scene.
  • the user can perform evidence search operations on the virtual props in the story scene, which operations include but are not limited to opening, closing, breaking, splicing, adjusting positions, tapping, etc., to obtain evidence on the virtual props. information.
  • the user can perform evidence collection operations on the NPC in the story scene, which operations include but are not limited to attacking, avoiding, touching, hugging, communicating, etc., to obtain evidence collection information on the NPC.
  • Step 268 Display second AR information in the AR video stream, where the second AR information is used to display evidence collection information of the first story character.
  • the evidence collection information obtained by User A in the virtual scene is displayed on the second AR information control in the virtual scene. If User A chooses to make it public, all story characters have viewing permissions. If User A chooses If private, no other story characters can view it.
  • the evidence collection information obtained by User B on the virtual prop is displayed on the second AR information control on the side of the virtual prop. If User B chooses to make it public, all story characters have viewing permission. If User B If you choose private, no other story characters can view it.
  • the evidence collection information of a certain story character obtained by user C during the evidence collection phase is displayed on the second AR information control on the side of the user. If user C chooses to make it public, all story characters have viewing permissions. , if user C chooses private, other story characters cannot view it.
  • Step 266a In the evidence collection stage, display the virtual scene related to the first story character
  • the user can freely select a virtual scene to conduct evidence collection.
  • a virtual scene For example, after entering the evidence collection stage, the user can freely select a virtual scene to conduct evidence collection.
  • user E wants to search for the bedroom of character four, and selects the bedroom of character four in the AR virtual scene displayed on the AR terminal.
  • the selection can be made by clicking, sliding, dragging, fixing the gaze for more than 5 seconds, etc. accomplish.
  • Step 266b In response to the evidence collection operation on the virtual scene, obtain the first evidence collection information of the first story character in the virtual scene;
  • user C searched for evidence in the fourth bedroom of the character and found that the furniture was messy, as if it was messed up by rummaging for something. After user C pointed the terminal at the messy furniture for 3 seconds, the terminal recognized and obtained to the evidence collection information.
  • Step 268a Display a second AR information control located around the real-life character corresponding to the first story character in the AR video stream, and the second AR information control displays evidence collection information of the first story character.
  • user C discovers that the furniture in the bedroom of character 4 bound by user D is messy due to rummaging for things.
  • This evidence search information is displayed on the AR information control on the side of user D.
  • the evidence collection information can be selected to be public or private. If it is selected to be public, other story characters in the story scene can also view it; if it is selected to be private, other story characters in the story scene cannot view it.
  • Step 266c In the evidence collection stage, display virtual props related to the first story character
  • the user's AR terminal can display virtual props related to a certain story character.
  • the virtual props exist in a specific virtual scene, or the virtual props do not need to exist in a specific virtual scene. in the scene.
  • user E chooses to search for evidence on the guqin of character 2 bound to user B.
  • User E can choose to view the guqin in any scene of the "Ancient Story One" reasoning task.
  • user E chooses to search for the dowry of character four bound to user D.
  • User E's terminal will display the dowry only after user E enters character four's bedroom.
  • Step 266d In response to the evidence search operation on the virtual prop, obtain the second evidence search information associated with the first story character on the virtual prop;
  • user E observes the AR virtual guqin of character 2 bound to user B, and finds a blood stain on the AR virtual guqin. After user E places his finger on the blood stain for 3 seconds, the AR terminal recognizes and obtains the blood stain. Information about character two.
  • user E searches for evidence in the AR virtual residence of character 4 bound to user D. He uses the AR terminal to point at character 4's makeup package for 5 seconds, opens the makeup package, and finds a damaged silver hairpin inside the makeup package. After user E pointed the AR terminal at the damaged silver hairpin for 3 seconds, the AR terminal recognized and obtained the evidence collection information about character four.
  • Step 268b In the AR video stream, display a second AR information control at the location where the evidence collection information is obtained, and the second AR information control displays the evidence collection information of the first story character.
  • user E found blood stains on the guqin of character 2 bound to user B.
  • This evidence search information is displayed on the AR information control on the side of the guqin.
  • user B finds a damaged silver hairpin in the dowry where character 4 lives, and this evidence search information is displayed on the AR information control on the side of the dowry.
  • the evidence collection information can be selected to be public or private. If it is selected to be public, other story characters in the story scene can also view it; if it is selected to be private, other story characters in the story scene cannot view it.
  • Step 266e In the evidence collection stage, display NPCs related to the first story character
  • the user's AR terminal can display an NPC related to a certain story character that exists in a specific virtual scene, or the NPC does not need to specifically exist in a specific virtual scene.
  • user B wants to search for the maid in the bedroom of character three, and selects the bedroom of character three in the AR virtual scene displayed on the AR terminal.
  • the selection can be made by clicking, sliding, dragging, or fixing the gaze for more than 5 seconds. It can be realized in seconds, and it will appear that there is a maid in the dormitory.
  • Step 266f In response to the interactive operation on the NPC story character, obtain the third evidence collection information associated with the first story character on the NPC story character;
  • user B searched for evidence from the maid in the bedroom of character three bound to user C. After pulling the sleeves of the maid, she found bruises. User B's terminal obtained from the server "Character 3 often beats the maid.” This evidence collection information.
  • Step 268c In the AR video stream, a second AR information control is displayed at the acquisition position where the NPC is located, and the second AR information control displays the evidence collection information of the first story character.
  • user B finds scars on the body of the maid in user C's character three, and this evidence search information is displayed on the AR information control on the side of user C.
  • the evidence collection information can be selected to be public or private. If it is selected to be public, other story characters in the story scene can also view it; if it is selected to be private, other story characters in the story scene cannot view it.
  • the user can use the terminal to complete the puzzle reasoning task based on the AR video stream.
  • the completion of the puzzle reasoning task based on the AR video stream provided in this embodiment can be divided into the following two situations:
  • the first case based on timeline control:
  • Step a Display the timeline control corresponding to the story scene, and display at least one of the character information and evidence collection information of the first story character in chronological order on the timeline control;
  • the timeline control is a control that can display character information or evidence collection information in chronological order.
  • the chronological order can be the real chronological order, or the chronological order of the story scenes.
  • user A sorts the acquired information about character two on the timeline control displayed on the AR terminal in chronological order in the story scene.
  • Step b In response to the reasoning operation on the timeline control, reason about the reasoning task corresponding to the story scene in the time dimension.
  • Inference operations based on the timeline control include but are not limited to finding or inferring doubtful points on the timeline in the obtained character information or evidence collection information.
  • user A finds evidence that character two's alibi at the time of the crime was invalid in the information of character two displayed in chronological order in the story scene.
  • the second case based on virtual map control:
  • Step c Display the virtual map control corresponding to the story scene, and display at least one of the character information and evidence collection information of the first story character according to the geographical location on the virtual map control;
  • the virtual map control is a control that can display character information or evidence collection information according to geographical location.
  • the virtual map can be a real geographical location, or the geographical location of a story scene.
  • user A displays the acquired information about character three on the virtual map control displayed on the AR terminal according to the geographical location in the story scene.
  • Step d In response to the reasoning operation on the virtual map control, reason about the reasoning task corresponding to the story scene in the spatial dimension.
  • Inference operations based on the virtual map control include but are not limited to finding or inferring doubtful points in the geographical location of the obtained character information or evidence collection information.
  • user A finds evidence that character three has been to the crime scene in the information of character three displayed according to the geographical location in the story scene.
  • Figure 11 shows a flow chart of a human-computer interaction method based on story scenarios provided by an exemplary embodiment of the present application. This method can be performed by both terminals and servers. The method includes:
  • Step 1101 The terminal obtains the live video stream collected by the camera;
  • the terminal is equipped with a camera. After starting the inference task, the camera on the terminal shoots the environment in front and obtains a real-life video stream.
  • the live video stream includes multiple live video frames (referred to as video frames) arranged in sequence.
  • the live video stream may be a video stream that has been encoded and compressed.
  • Step 1102 The terminal sends the live video stream to the server;
  • the terminal sends the live video stream to the server through the wireless network or wired network.
  • Step 1103 The server receives the live video stream reported by the terminal;
  • Step 1104 The server performs image semantic recognition on the real-scene video frames in the real-scene video stream, and obtains the background area and foreground character area in the real-scene video frame; the foreground character area corresponds to the real-scene character;
  • the image semantic segmentation model is stored in the server.
  • the server inputs the semantic segmentation result of the previous video frame and the current video frame in the live video stream into the image semantic segmentation model to obtain the semantic segmentation result of the current video frame.
  • the semantic segmentation result includes the background area and the foreground character area.
  • the server when processing the first video frame in the live video stream, the server will input the reference segmentation result and the first video frame into the image semantic segmentation model to obtain the semantic segmentation result of the first video frame.
  • the reference segmentation result may be a preset segmentation result, or a rough segmentation result after semantic segmentation of the first video frame using other models, or the reference segmentation result may be a blank segmentation result, which is not limited by this application. .
  • the server when processing other video frames except the first video frame in the real-life video stream, the server inputs the segmentation result of the i-1 video frame and the i-th video frame into the image semantic segmentation model. , obtain the semantic segmentation result of the i-th video frame.
  • the image semantic segmentation model can output 2 semantic categories: background area and foreground person area; in one example, the image semantic segmentation model can output 3 semantic categories: background area, face in foreground person area. facial area and non-face area; in another example, the image semantic segmentation model can output multiple semantic categories: multiple sub-areas in the background area and the foreground person area, the multiple sub-areas include the face area, torso area, At least two of the limb areas, palm areas, finger areas, and backbone key points. This application does not limit the semantic segmentation capabilities of the image semantic segmentation model.
  • Step 1105 The server processes the picture content in the background area to obtain an augmented reality AR background area, and processes the picture content in the foreground character area to obtain an AR character area;
  • the AR background area displays the scene picture of the story scene, and the AR character area displays the wearable Real-life characters dressed up in AR, and the AR clothes correspond to the story characters in the story scenes;
  • Each story scene corresponds to scene materials, and each story character corresponds to character materials.
  • the scene materials of the story scene include but are not limited to: at least one of natural environment materials, humanistic architectural materials, outdoor decoration materials, indoor decoration materials, furniture materials, and environmental prop materials.
  • Character materials for story characters include but are not limited to: jewelry materials, facial makeup materials, top materials, pants materials, dress materials, shoe materials, hand-held prop materials, vehicle materials or mount materials, etc.
  • the character materials of an ancient swordsman story include hosta materials, sect costumes, sword materials, etc.
  • the character materials of a western cowboy story include cowboy hat materials, shirt materials, jeans materials, horse materials, pistol materials, etc.
  • the server After identifying the background area and the foreground character area in the live video frame, the server replaces or fuses the background area based on the scene material of the story scene to obtain the AR background area; and the non-face area based on the character material of the first story character Replace or fuse to get the AR character area.
  • the server directly uses scene materials to replace the background area to obtain an AR background area without considering any physical information in the background area, such as replacing the office background in the real scene with the background of the martial arts mountain; in another
  • the server takes into account the three-dimensional structural information in the background area and retains the original main body structure in the background area. It uses the surface map in the scene material to re-render the environment in the background area to obtain an image based on the original main body.
  • the structure's personalized AR background area such as re-rendering a room in the real scene into a royal concubine's living room.
  • the server replaces the clothing in the non-face area to obtain an AR character area based on the character material of the first story character; or, the server adds virtual jewelry or virtual props to the non-face area based on the character material of the first story character. , get the AR character area.
  • the story role bound to each live-action character can be different.
  • the same story character has different AR costumes in different time periods of the same story scene; or the same story character has different AR costumes in different locations of the same story scene; or the same story Characters have different AR costumes in different story scenes; or the same story character has different AR costumes in different time periods of different story scenes; or the same story character has different AR costumes in different locations in different story scenes.
  • Step 1106 The server obtains an AR video stream based on the AR video frame that combines the AR background area and the AR character area;
  • AR video frames there is a one-to-one correspondence between AR video frames and real scene video frames.
  • the processed AR background area and AR character area of the same real-scene video frame are combined to obtain an AR video frame corresponding to the real-scene video frame.
  • the server splices each AR video frame in chronological order to obtain an AR video stream.
  • the server also encodes and compresses the AR video stream to reduce network bandwidth usage during data transmission.
  • Step 1107 The server sends the AR video stream to the terminal;
  • the server sends the live video stream to the terminal through the wireless network or wired network.
  • Step 1108 The terminal completes the reasoning task corresponding to the story scene based on the AR video stream.
  • the method provided by this embodiment can significantly reduce the local computing resource consumption of the terminal and provide a smoother AR experience by allowing the server located in the cloud to undertake the computing task of image semantic segmentation.
  • this embodiment implements dynamic video semantic segmentation based on traditional static semantic segmentation.
  • traditional static semantic segmentation the image to be classified is input into the image semantic segmentation model for semantic segmentation, and the semantic segmentation result is obtained.
  • the semantic segmentation result of the previous video frame and the current video frame are input into the image semantic segmentation model for semantic segmentation, and the semantic segmentation result of the current video frame is obtained. Since in the image semantic segmentation task of the current video frame, the semantic segmentation result of the previous video frame is introduced as reference information, the high correlation between the previous video frame and the current video frame in the time domain can be used to speed up the image semantic segmentation task. The calculation speed shortens the time-consuming when performing image semantic segmentation on the current video frame, and improves the accuracy when performing image semantic segmentation on the current video frame.
  • the image semantic segmentation model is trained based on a basic sample library.
  • the basic sample library includes: the semantic segmentation label of the previous sample video frame, the current sample video frame, and the semantic segmentation label of the current sample video frame.
  • the previous sample video frame is the video frame located before the current sample video frame in the sample video frame. If the current sample video frame is the first frame, the previous video frame can use the current sample after affine transformation or thin plate interpolation video frames instead.
  • the semantic segmentation identifier of the previous sample video frame and the semantic segmentation label of the current sample video frame can be a manually annotated cutout sample mask, or a cutout sample mask after semantic segmentation by a traditional static image semantic segmentation model.
  • Affine transformation It is an image transformation method that describes a linear transformation of two-dimensional coordinate points. Affine transformation can perform a linear transformation on a two-dimensional coordinate point followed by a translation. In the embodiment of the present application, affine transformation can simulate the movement of real characters.
  • Thin Plate Spline (TPS, also known as Thin Plate Spline Interpolation): It is a two-dimensional interpolation method that is used to offset control points on the image to achieve specific deformation of the image through the control points. the goal of. In the embodiment of this application, the thin plate insertion can simulate the rapid shaking of the camera.
  • the image semantic segmentation model is trained based on a basic sample library and an enhanced sample library
  • the basic sample library includes: the semantic segmentation label of the previous sample video frame, the current sample video frame, and the semantic segmentation label of the current sample video frame;
  • the enhanced sample library includes: the semantic segmentation label of the previous sample video frame, the current enhanced video frame, and the semantic segmentation label of the current enhanced video frame;
  • the current enhanced video frame is obtained by performing affine transformation or thin plate interpolation on the current sample video frame
  • the semantic segmentation label of the current enhanced video frame is obtained by performing affine transformation and/or thin plate interpolation on the semantic segmentation label of the current sample video frame. Got it this way.
  • the server performs the same affine transformation or thin plate interpolation on the background area in the semantic segmentation label of the same pair of current sample video frames and the current sample video frame to obtain the first enhanced sample; the server performs the same pair of current sample video frames on the background area in the semantic segmentation label. After the foreground character area in the semantic segmentation label of the frame and the current sample video frame is subjected to the same affine transformation or thin plate interpolation, the second enhanced sample is obtained.
  • the above image semantic segmentation model can be implemented using a fully convolutional network (Fully Convolutional Networks, FCN) network.
  • FCN Fully Convolutional Networks
  • the image semantic segmentation model needs to determine the category of each pixel in the image. That is, image semantic segmentation is at the pixel level.
  • CNN Convolutional Neural Network
  • the overall network structure of FCN is divided into two parts: the full convolution part and the deconvolution part.
  • the full convolution part borrows some classic CNN networks (such as AlexNet, VGG, GoogLeNet, etc.).
  • AlexNet is a neural network launched in 2012.
  • VGG refers to the Visual Geometry Group network.
  • GoogLeNet was proposed in 2014.
  • a new deep learning structure and replaces the final fully connected layer with a convolutional layer, which is used to extract features to form a heat map; the deconvolution part is to upsample the small-sized heat map to obtain the original Dimensional semantic segmentation of images.
  • the input of the FCN network can be a color image of any size, the output is the same size as the input, and the number of channels is n (number of target categories) + 1 (background).
  • the FCN network does not use a fully connected layer in the convolutional part of the CNN but replaces it with a convolutional layer. The purpose is to allow the input image to be any size beyond a certain size.
  • the predictions (FCN-32s) of the bottom layer (step size 32) are upsampled by 2 times to obtain the original size image, and fused (added) with the predictions from the pool 4 layer (step size 16). This part The network is called FCN-16s. This part of the prediction is then upsampled by a factor of 2 and fused with the predictions from pool 3. This part of the network is called FCN-8s. With reference to Figure 14, Figure 14 shows the effect comparison between FCN-32s, FCN-16s, FCN-8s and real samples.
  • FCN classifies images at the pixel level, thereby solving the semantic-level image segmentation problem.
  • FCN can accept input images of any size and uses a deconvolution layer to upsample the feature map of the last convolutional layer. It is restored to the same size of the input image, so that a prediction can be generated for each pixel, while retaining the spatial information in the original input image, and finally pixel-by-pixel classification is performed on the upsampled feature map.
  • the classification loss is calculated pixel by pixel, which is equivalent to each pixel corresponding to a training sample.
  • Figure 15 shows an AR screen 300 displayed by the AR terminal of user E during the evidence collection phase.
  • the screen includes an AR background area 301, an AR character area 302 and an AR information control.
  • the AR background area 301 displays the virtual scene and virtual props 305 in the evidence collection stage.
  • the AR character area 302 displays real people wearing AR costumes 303.
  • the AR costumes 303 correspond to the story characters in the story scene.
  • the AR information control 304 displays the characters of the story characters.
  • Information or evidence collection information can be located around the real-life characters bound to the story characters or at the location where the evidence collection information is obtained.
  • the AR background area 301 is displayed under a willow tree at the foot of a mountain outside the city.
  • the AR character area 302 displays user B wearing antique AR clothing, and user B is playing the AR virtual guqin.
  • the character information of the second story character that user E has acquired is displayed on the AR information control 304 on the side of user B.
  • the private information of the second story character obtained by user E cannot be viewed by user A, user B, user C, and user D.
  • the public information of the second story character obtained by user E cannot be viewed by user A, user B, user C, and user D. Can be viewed.
  • the evidence search information of the second story character obtained by user E after the virtual Guqin evidence search operation is displayed on the AR information control located at the location where the evidence search information is obtained.
  • the information can be selected to be public or private. If it is selected to be public, the story Other story characters in the scene can also view it; if private is selected, other story characters in the story scene cannot view it.
  • Figure 16 shows a game scene flowchart of a story scene-based human-computer interaction method provided by an exemplary embodiment of the present application.
  • the method is executed by the terminal shown in FIG. 3 as an example.
  • the terminal has a camera.
  • the method includes: selecting reasoning tasks 1601, selecting story characters 1602, reading scripts 1603, introduction stage 1604, public chat stage 1605, private chat stage 1606, evidence collection stage 1607 and case closing stage 1608.
  • Select reasoning task 1601 The user selects any reasoning task among the reasoning tasks of at least two candidate story scenes displayed on the terminal;
  • user A selects the gunfight and spy scene among the gunfight and spy scene, the fairy cultivating scene, the western cowboy scene, and the tomb exploration scene.
  • Select story character 1602 After the user selects the story scene, he selects any story character from at least two candidate story characters displayed on the terminal.
  • the terminal camera collects the user's image and performs face recognition, performs AR dress-up on the user, and connects the story character with the user. match;
  • user A selects the first story character Agent A and completes character binding and AR dress change.
  • Reading script 1603 The user reads the background information of the selected story scene, and understands the background, time, tasks and basic information of the bound story characters, etc.;
  • Introduction stage 1604 The user introduces himself to other story characters in the same story scene and obtains basic information about other story characters in the same story scene. This information is public information.
  • the information acquisition methods include: obtaining from the server and voice input. , OCR scanning and keyboard input, the information is displayed in the AR information control, and the user can scan the corresponding story character through the terminal device to view the AR information;
  • user A introduces himself to users in the same story scene, and obtains the basic information of user B and user C from the server. This information is displayed in the AR information control located next to user B and user C.
  • the information is public information.
  • User A, User B and User C can scan the corresponding story characters through their terminal devices to view their basic information.
  • Public chat stage 1605 All users in the same story scene exchange information. Users can obtain extended information about story characters in the same story scene. This information is public information. Information acquisition methods include: obtaining from the server, voice input, At least one of OCR scanning and keyboard input, the information is displayed in the AR information control, and the user can scan the corresponding story character through the terminal device to view the AR information;
  • user A obtains the schedules of user B and user C for the past three days through OCR scanning during the public chat stage.
  • This information is displayed in the AR information control located on the side of user B and user C.
  • This information is public.
  • Information, User A, User B and User C can use the terminal device to scan the story roles bound to User B and User C to view the extended information.
  • Private chat stage 1606 Only two story characters in the same story scene exchange information. The user obtains extended information about the story character with whom he has a private chat. This information is private information.
  • the information acquisition methods include: obtaining from the server and voice input. , OCR scanning and keyboard input, the information is displayed in the AR information control, and the user can scan the corresponding story character through the terminal device to view the AR information;
  • user A chats privately with user B and obtains the extended information of the second story character bound by user B through text input: the tool source of user B is user C, and the information is displayed on the icon located next to user B.
  • the AR information control only user A has permission to view this information.
  • User A can scan the second story role bound to user B through the terminal device to view the extended information.
  • Evidence collection stage 1607 The story character conducts evidence collection operations on virtual scenes or virtual props related to other story characters in the same story scene to obtain evidence collection information of other story characters. This information can be made public or private. If it is made public, Then other story characters in the story scene can also view it; if you select private, other story characters in the story scene cannot view it. Users can scan the corresponding story characters or virtual props through the terminal device to view the evidence collection information.
  • user A conducts an evidence search operation on the desk of the third story character bound by user C, and obtains a tool buying and selling list. This information is displayed on the AR information control located at the desk, and user A chooses to disclose the evidence search information.
  • User A, User B and User C all use the terminal device to scan the third story character or the desk bound to User C to view the evidence collection information.
  • Case closing stage 1608 User A, User B and User C vote.
  • the voting result is that User B is the target person.
  • the inference result is correct.
  • the inference task is completed and the case is closed.
  • the method provided by this embodiment uses the terminal to perform human-computer interaction and reasoning tasks, uses face recognition and AR dress-up to bind the user to the story character, and can obtain it from the server, voice input, OCR scanning and keyboard Input at least one of the above to obtain character information and evidence collection information.
  • the game operation is simple and convenient, providing a more immersive gaming experience.
  • Figure 17 shows a structural block diagram of a computer system provided by another exemplary embodiment of the present application. This embodiment illustrates that the method is executed by the computer system shown in FIG. 3 .
  • the system includes: client 1701, background service 1702, architecture engine 1703, data storage 1704, and running environment 1705.
  • Client 1701 refers to the Android or iOS application on the terminal that supports AR interaction and reasoning tasks.
  • the terminal can be an electronic device with a camera such as a smartphone, a tablet computer, an e-book reader, a laptop computer, a desktop computer, and AR glasses.
  • the client 1701 supports the terminal to perform script selection operations, story character selection operations, and facial information entry.
  • the client 1701 supports at least one AR function of displaying AR scenes, AR clothing, and AR information.
  • the client 1701 supports the information recording function and can record information through at least one of OCR input, voice input and keyboard input;
  • Background service 1702 a background service provided by the server 320 that supports at least one of the data services, AR services and intelligent input services executed by the client 1701, intercepts and responds to requests from the client 1701, and responds to all requests from the client 1701 Screen, filter or call third-party interfaces, package the information and then send it back to the client 1701;
  • Architecture engine 1703 Perform operations such as starting applications, processing request parameters, and rendering response formats through the GIN framework (a web page framework), processing operations of AR functions through the AR engine, and processing calculations related to machine learning through the AI engine. operate;
  • GIN framework a web page framework
  • Data storage 1704 includes the MySQL database (a relational database management system) that stores general information and the MongoDB database (a database based on distributed file storage) that stores massive user logs and user galleries.
  • MySQL database a relational database management system
  • MongoDB database a database based on distributed file storage
  • the two databases are stored independently. All implement cluster distributed deployment storage through Hadoop (which is a distributed system infrastructure), and use Distributed Relational Database Service (DRDS) as middleware to achieve elastic storage;
  • DRDS Distributed Relational Database Service
  • Running environment 1705 The background service 1702 uses the cloud computing platform to undertake the training tasks of the discriminator and generator based on the client data set, replaces the real-life video stream with an AR video stream through face recognition and image semantic recognition, and then transmits the image back Android or iOS client 1701 that supports AR interaction and reasoning tasks provides users with a smoother and more immersive AR experience.
  • Figure 18 shows a block diagram of a human-computer interaction device based on a story scene provided by an exemplary embodiment of the present application.
  • the device includes: an acquisition module 1802, a display module 1804, a processing module 1806, and an interaction module 1808.
  • Obtaining module 1802 used to perform step 220 shown in Figure 2 in the above embodiment.
  • Display module 1804 used to perform step 240 shown in Figure 2 in the above embodiment.
  • the display module 1804 is used to display first AR information in the AR video stream, and the first AR information is used to display character information of the first story character.
  • the display module 1804 is used to display second AR information in the AR video stream, and the second AR information is used to display evidence collection information of the first story character.
  • the display module 1804 is used to display a first AR information control located around the real person in the AR video stream, and the first AR information control displays the first AR information.
  • the display module 1804 is used to display the second AR information control in the AR video stream, and the second AR information control is used to display the second AR information.
  • the display module 1804 is configured to display a second facial image of the real person without the AR device in the second facial area of the AR character area.
  • Processing module 1806 used to perform at least one of steps 241 to 247 shown in Figure 2 in the above embodiment.
  • Interaction module 1808 used to execute step 260 shown in Figure 2, step 266a-step 268a shown in Figure 8, step 266c-step 268b shown in Figure 9, and step 266e-step shown in Figure 10 in the above embodiment. At least one step in 268c.
  • the interaction module 1808 is used for at least one of the following methods: obtaining the character information of the first story character from the server; obtaining the character information of the first story character through voice input; using optical character recognition OCR The character information of the first story character is obtained through scanning; the character information of the first story character is obtained through keyboard input.
  • the device further includes: an upload module, configured to receive an upload operation of the AR costume; in response to the upload operation, upload the locally created AR costume to the server.
  • an upload module configured to receive an upload operation of the AR costume; in response to the upload operation, upload the locally created AR costume to the server.
  • the device also includes: a custom module for receiving a custom operation of the AR costume; in response to the custom operation, uploading the customized AR costume to the server.
  • Figure 19 shows a block diagram of a human-computer interaction device based on a story scene provided by an exemplary embodiment of the present application.
  • the device includes: a receiving module 1902, a processing module 1904, and an interactive module 1906.
  • the receiving module 1902 is used to perform step 1103 shown in Figure 11 in the above embodiment.
  • the processing module 1904 is configured to perform at least one of steps 1104 to 1106 shown in Figure 11 in the above embodiment.
  • the interactive module 1906 is used to complete reasoning tasks corresponding to story scenes based on AR video streams.
  • the interaction module 1906 completes at least one of an information acquisition task, an evidence collection task, and a puzzle reasoning task corresponding to the story scene.
  • the interactive module 1906 is used to obtain the role information of the first story character; in an optional design, the interactive module 1906 is used to obtain evidence collection information of the first story character.
  • the interactive module 1906 is used to search for reasoning tasks corresponding to the story scenes in the time dimension in response to the reasoning operation on the timeline control; or, the interactive module 1906 is used to respond to the virtual map control
  • the inference operation on the virtual scene is used to search for the inference task corresponding to the story scene in the spatial dimension; or, the interaction module 1906 is used to obtain the first story character's third position in the virtual scene in response to the viewing operation of the specified position in the virtual scene.
  • An evidence search information or, the interaction module 1906 is used to obtain the second evidence search information associated with the first story character on the virtual prop in response to the interactive operation on the virtual prop; or, the interaction module 1906 is used to respond to the NPC story
  • the interactive operation of the character obtains the third evidence collection information of the character in the first story.
  • Figure 20 shows a structural block diagram of a terminal 2000 provided by an exemplary embodiment of the present application.
  • the terminal 2000 may be an electronic device with a camera such as a smartphone, a tablet computer, an e-book reader, a laptop computer, a desktop computer, and AR glasses.
  • the terminal 2000 may also be called a user equipment, a portable terminal, a laptop terminal, a desktop terminal, and other names.
  • the terminal 2000 includes: a processor 2001 and a memory 2002.
  • the processor 2001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc.
  • the processor 2001 can adopt at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), and PLA (Programmable Logic Array, programmable logic array).
  • the processor 2001 may also include a main processor and a co-processor.
  • the main processor is a processor used to process data in a wake-up state, also called a CPU; a co-processor is used to process data in a standby state. Low-power processor for processing.
  • the processor 2001 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is responsible for rendering and drawing content that needs to be displayed on the display screen.
  • the processor 2001 may also include an AR processor, which is used to process computing operations related to augmented reality.
  • the processor 2001 may also include an AI (Artificial Intelligence, artificial intelligence) processor, which is used to process computing operations related to machine learning.
  • AI Artificial Intelligence, artificial intelligence
  • Memory 2002 may include one or more computer-readable storage media, which may be non-transitory. Memory 2002 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 2002 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 2001 to implement the story-based method provided by the method embodiments in this application. Scenario-based human-computer interaction methods.
  • the terminal 2000 optionally further includes: a peripheral device interface 2003 and at least one peripheral device.
  • the processor 2001, the memory 2002 and the peripheral device interface 2003 may be connected through a bus or a signal line.
  • Each peripheral device can be connected to the peripheral device interface 2003 through a bus, a signal line or a circuit board.
  • the peripheral device may include: at least one of a radio frequency circuit 2004, a display screen 2005, a camera assembly 2006, an audio circuit 2007, and a power supply 2008.
  • the peripheral device interface 2003 may be used to connect at least one I/O (Input/Output, input/output) related peripheral device to the processor 2001 and the memory 2002 .
  • I/O Input/Output, input/output
  • the radio frequency circuit 2004 is used to receive and transmit RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals.
  • RF Radio Frequency, radio frequency
  • the display screen 2005 is used to display UI (User Interface, user interface).
  • UI User Interface, user interface
  • the camera component 2006 is used to collect images or videos.
  • Audio circuit 2007 may include a microphone and speakers.
  • the power supply 2008 is used to provide power to various components in the terminal 2000.
  • the terminal 2000 further includes one or more sensors 2009.
  • the one or more sensors 2009 include, but are not limited to: acceleration sensor 2010, gyro sensor 2011, pressure sensor 2012, optical sensor 2013, and proximity sensor 2014.
  • the acceleration sensor 2010 can detect the acceleration on the three coordinate axes of the coordinate system established by the terminal 2000 .
  • the gyro sensor 2011 can detect the body direction and rotation angle of the terminal 2000, and the gyro sensor 2011 can cooperate with the acceleration sensor 2011 to collect the user's 3D movements on the terminal 2000.
  • the pressure sensor 2012 may be provided on the side frame of the terminal 2000 and/or on the lower layer of the display screen 2005 .
  • the optical sensor 2013 is used to collect ambient light intensity.
  • the proximity sensor 2014 also called a distance sensor, is usually provided on the front panel of the terminal 2000.
  • the proximity sensor 2014 is used to collect the distance between the user and the front of the terminal 2000.
  • the memory also includes one or more programs, the one or more programs are stored in the memory, and the one or more programs include a method for performing the signal display method based on the virtual environment provided by the embodiment of the present application.
  • FIG. 20 does not constitute a limitation on the terminal 2000, and may include more or fewer components than shown, or combine certain components, or adopt different component arrangements.
  • a terminal is also provided.
  • the terminal includes a processor and a memory, and at least one instruction, at least a program, a code set or an instruction set are stored in the memory.
  • the at least one instruction, at least one program, code set or instruction set is configured to be executed by the processor to implement the above-mentioned human-computer interaction method based on story scenarios.
  • a server 2100 is also provided, and the server 2100 includes a processor 2101 and a memory 2102.
  • Figure 21 shows a structural block diagram of a server 2100 provided by an exemplary embodiment of the present application.
  • the server 2100 includes: a processor 2101 and a memory 2102.
  • the processor 2101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc.
  • the processor 2101 can adopt at least one hardware form among digital signal processing (Digital Signal Processing, DSP), field-programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA).
  • DSP Digital Signal Processing
  • FPGA field-programmable gate array
  • PROM programmable logic array
  • PLA programmable logic array
  • the processor 2101 can also include a main processor and a co-processor.
  • the main processor is a processor used to process data in the wake-up state, also called a central processing unit (Central Processing Unit, CPU); the co-processor is A low-power processor used to process data in standby mode.
  • CPU Central Processing Unit
  • the processor 2101 may be integrated with a graphics processor (Graphics Processing Unit, GPU), and the GPU is responsible for rendering and drawing content that needs to be displayed on the display screen.
  • the processor 2101 may also include an artificial intelligence (Artificial Intelligence, AI) processor, which is used to process computing operations related to machine learning.
  • AI Artificial Intelligence
  • Memory 2102 may include one or more computer-readable storage media, which may be non-transitory. Memory 2102 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 2102 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 2101 to implement the three-dimensional space provided by the method embodiments in this application. global illumination calculation method.
  • the server 2100 optionally further includes: an input interface 2103 and an output interface 2104.
  • the processor 2101, the memory 2102, the input interface 2103, and the output interface 2104 may be connected through a bus or a signal line.
  • Each peripheral device can be connected to the input interface 2103 and the output interface 2104 through a bus, a signal line or a circuit board.
  • the input interface 2103 and the output interface 2104 may be used to connect at least one peripheral device related to input/output (I/O) to the processor 2101 and the memory 2102 .
  • the processor 2101, the memory 2102, the input interface 2103, and the output interface 2104 are integrated on the same chip or circuit board; in some other embodiments, the processor 2101, the memory 2102, the input interface 2103, and the output interface Any one or two of 2104 can be implemented on a separate chip or circuit board, which is not limited in the embodiment of the present application.
  • server 2100 does not constitute a limitation on the server 2100, and may include more or fewer components than shown, or combine certain components, or adopt different component arrangements.
  • a computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program
  • the above-mentioned computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory, CD-ROM) ), tapes, floppy disks and optical data storage devices, etc.
  • a computer program product stores a computer program, and the computer program is loaded and executed by the processor to implement the human-computer interaction method based on the story scene as described above. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种基于故事场景的人机互动方法、装置、设备及介质，应用于增强现实领域中，方法由具有摄像头的终端（310，330）执行。方法包括：获取摄像头采集的实景视频流（220），基于实景视频流显示AR视频流（240），响应于交互操作，改变AR视频流的显示内容（260），基于改变后的显示内容，完成故事场景对应的推理任务（280）。由于AR背景区域是对实景视频流中的背景区域处理得到的，AR人物区域是对实景视频流中的前景人物区域处理替换得到的，使得无需借助实物的服装、道具和纸件剧本，就提供出非常有沉浸感的视觉效果，在减少物理资源的耗费的情况下，实现更加优秀的视觉体验。

Description

基于故事场景的人机互动方法、装置、设备及介质
本申请要求于2022年04月18日提交的申请号为202210406828.1、发明名称为“基于故事场景的人机互动方法、装置、设备及介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及增强现实(Augmented Reality，AR)领域，特别涉及一种基于故事场景的人机互动方法、装置、设备及介质。
背景技术
剧本杀是多个玩家分别扮演一个故事角色，每个故事角色分别进行推理流程的游戏。
相关技术提供的线下剧本杀技术中，采用投影仪在剧本杀房间的墙壁上投射投影画面。该投影画面用于模拟指定的剧本杀场景，比如，古装婚礼场景、民国谍战场景等。玩家需要更换指定的服装和道具，使用纸件剧本在剧本杀房间内完成推理。
虽然上述线下剧本杀已经具有较好的沉浸式体验，但是实现该种沉浸式体验需要耗费较多的物理资源，比如为剧本杀场景专门定制的服装、道具、纸件剧本等。在线下剧本杀的剧本较多的情况下，这种专门定制的物理资源的耗费更为严重。
发明内容
本申请提供了一种基于故事场景的人机互动方法、装置、设备及介质。技术方案如下：
根据本申请的一个方面，提供了一种基于故事场景的人机互动方法，该方法由具有摄像头的终端执行，方法包括：
获取摄像头采集的实景视频流，实景视频流的视频画面包括背景区域和前景人物区域，前景人物区域对应实景人物；
基于所述实景视频流显示AR视频流，所述AR视频流的视频画面包括AR背景区域和AR人物区域，所述AR背景区域显示所述故事场景的场景画面，所述AR人物区域显示穿戴有AR装扮的所述实景人物，所述AR装扮对应所述故事场景中的故事角色；
响应于交互操作，改变所述AR视频流的显示内容；
基于改变后的显示内容，完成所述故事场景对应的推理任务；
其中，所述AR背景区域是对所述背景区域中的画面内容处理得到的，所述AR人物区域是对所述前景人物区域中的画面内容处理得到的。
根据本申请的一个方面，提供了一种基于故事场景的人机互动方法，方法包括：
接收终端上报的实景视频流；
对实景视频流中的实景视频帧进行图像语义识别，得到实景视频帧中的背景区域和前景人物区域；前景人物区域对应实景人物；
对背景区域中的画面内容处理得到AR背景区域，以及对前景人物区域中的画面内容处理得到AR人物区域；AR背景区域显示故事场景的场景画面，AR人物区域显示穿戴有AR装扮的实景人物，AR装扮对应故事场景中的故事角色；
基于将AR背景区域和AR人物区域合并后的AR视频帧，得到AR视频流；
向终端发送AR视频流，以便终端基于AR视频流完成故事场景对应的推理任务。
根据本申请的一个方面，提供了一种基于故事场景的人机互动装置，装置包括：
获取模块，用于获取摄像头采集的实景视频流，实景视频流的视频画面包括背景区域和前景人物区域，前景人物区域对应实景人物；
显示模块，用于基于所述实景视频流显示AR视频流，所述AR视频流的视频画面包括AR背景区域和AR人物区域，所述AR背景区域显示所述故事场景的场景画面，所述AR人物区域显示穿戴有AR装扮的所述实景人物，所述AR装扮对应所述故事场景中的故事角色；
互动模块，用于响应于交互操作，改变所述AR视频流的显示内容；基于改变后的显示内容，完成所述故事场景对应的推理任务；
其中，所述AR背景区域是对所述背景区域中的画面内容处理得到的，所述AR人物区域是对所述前景人物区域中的画面内容处理得到的。
根据本申请的一个方面，提供了一种基于故事场景的人机互动装置，装置连接有摄像头，装置包括：
接收模块，用于接收摄像头采集的实景视频流，实景视频流的视频画面包括背景区域和前景人物区域，前景人物区域对应实景人物；
处理模块，用于将实景视频流处理为增强现实AR视频流，AR视频流的视频画面包括AR背景区域和AR人物区域，AR背景区域显示故事场景的场景画面，AR人物区域显示穿戴有AR装扮的实景人物，AR装扮对应故事场景中的故事角色；
互动模块，用于基于AR视频流完成故事场景对应的推理任务；
其中，AR背景区域是对背景区域中的画面内容处理得到的，AR人物区域是对前景人物区域中的画面内容处理得到的。
根据本申请的另一方面，提供了一种终端，终端包括：处理器和存储器，存储器存储有计算机程序，计算机程序由处理器加载并执行以实现如上的基于故事场景的人机互动方法。
根据本申请的另一方面，提供了一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，计算机程序由处理器加载并执行以实现如上的基于故事场景的人机互动方法。
根据本申请的另一方面，提供了一种计算机程序产品，计算机程序产品存储有计算机程序，计算机程序由处理器加载并执行以实现如上的基于故事场景的人机互动方法。
本申请实施例提供的技术方案带来的有益效果至少包括：
通过将实景视频流中的背景区域替换为推理任务对应AR背景区域，将实景视频流中的实景人物替换为穿戴有AR装扮的实景人物，从而实现采用AR场景和AR装扮来营造剧本杀或密室逃脱的故事场景，无需借助专门定制的服装、道具和纸件剧本，就能够提供出非常有沉浸感的视觉效果。在减少专门定制的物理资源的耗费的情况下，实现比相关技术更加优秀的视觉体验。
附图说明
图1示出了本申请一个示例性实施例提供的一个用户通过摄像头采集到的实景视频流的示意图；
图2示出了本申请一个示例性实施例提供的一个用户的AR终端显示的AR视频流的示意图；
图3示出了本申请一个示例性实施例提供的计算机系统的结构框图；
图4示出了本申请一个示例性实施例提供的基于故事场景的人机互动方法的流程图；
图5示出了本申请一个示例性实施例提供的基于实景视频流显示AR视频流的方法的流程图；
图6示出了本申请一个示例性实施例提供的在AR视频流中不显示实景人物佩戴AR眼镜的示意图；
图7本申请另一个示例性实施例提供的基于故事场景的人机互动的方法的流程图；
图8示出了本申请一个示例性实施例提供的基于故事场景获取第一故事角色的搜证信息的方法的流程图；
图9示出了本申请另一个示例性实施例提供的基于故事场景获取第一故事角色的搜证信息的方法的流程图；
图10示出了本申请另一个示例性实施例提供的基于故事场景获取第一故事角色的搜证信息的方法的流程图；
图11示出了本申请一个示例性实施例提供的基于故事场景的人机互动方法的流程图；
图12示出了本申请一个示例性实施例提供的动态视频语义分割的示意图；
图13示出了本申请一个示例性实施例提供的FCN网络结构的示意图；
图14示出了本申请一个示例性实施例提供的FCN语义分割结果与真实样本效果对比的示意图；
图15示出了本申请一个示例性实施例提供的搜证阶段一个用户的AR终端显示的AR画面的示意图；
图16示出了本申请一个示例性实施例提供的基于故事场景的人机互动方法的游戏场景流程图；
图17示出了本申请另一个示例性实施例提供的计算机系统的结构框图；
图18示出了本申请一个示例性实施例提供的基于故事场景的人机互动的装置的框图；
图19示出了本申请另一个示例性实施例提供的基于故事场景的人机互动的装置的框图；
图20示出了本申请一个示例性实施例提供的终端的结构框图；
图21示出了本申请一个示例性实施例提供的服务器的结构框图。
具体实施方式
首先，对本申请实施例中涉及的名词进行简单介绍：
推理任务：又称推理游戏，是一名或多名玩家根据故事场景中的线索解决谜题的任务。传统的故事场景主要靠纸质媒介上的文字来营造。在近几年线下火热的剧本杀游戏和密室逃脱游戏中，故事场景由商家搭建的游戏场地来营造。
剧本杀：起源于一类真人角色扮演游戏，原型叫做谋杀之谜(Mystery of Murder)，游戏以剧本为核心，游戏过程中由主持人(DM)引导推动游戏进度，玩家通过多轮的搜证、发言、推理环节来完成自己的故事角色的推理任务，并还原事件发生的过程(作案手法)。例如在某个剧本中，故事角色需要破解还原凶手在密室杀人的手法。通常剧本中各个故事角色的人物关系也错综关联，玩家需要沉浸在故事角色当中，以及根据在场玩家的发言和信息量仔细推敲，最终投出自己认定的凶手，游戏结束后由主持人揭露真相并进行游戏复盘。有些剧本还会根据玩家的选择触发多种结局中的某个结局，这种存在多种结局的剧本称为“机制本”。
整个游戏流程大致如下：
·主持人给各个玩家分发不同故事角色的剧本；
·机制本会进行道具的分发，该道具用于供玩家选择以触发多种结局中的某个结局；
·玩家根据各个故事角色进行自我介绍；
·在主持人的引导下，玩家逐步阅读剧本，比如第1幕、第2幕，以此类推；
·搜证环节及机制触发环节；
·由各个玩家投票选出凶手；
·由主持人进行剧情复盘-游戏结束；
·通常情况下1个游戏的时间为4-5个小时。
密室逃脱：一种实景逃脱类游戏，最早的真人密室逃脱起源于2006年根据小说灵感设计了一系列的场景，并把它们还原到现实中，提供给所有员工进行冒险解谜，命名为“origin”。游戏的主要创意多源于影视剧、书籍和网络等场景，游戏过程中，玩家通常采用第一视角或以第三视角扮演主人公的模式下，被限定在一个近乎完全封闭或对自身存在威胁的环境(即“密室”)内，单个游戏存在至少一个密室，玩家需要发现和利用身边的道具(如纸质道具、机械道具、电子道具、真人扮演道具等)，推理并完成指定任务(通常是解开特定谜题的方式)，最终达到逃离该区域的目的。
故事场景：每个推理任务都会对应一个故事，该故事所处的时间、地点和环境构成了该推理任务的故事场景。比如，民国谍战场景、仙侠修仙场景、西部牛仔场景和古墓探险场景等。
故事角色：每个推理任务都存在至少一个人物角色，不同的人物角色具有不同的性别、不同的形象、不同的性格、故事背景、剧情推动作用以及推理任务。该故事角色可以是虚拟人物、虚拟动物、动漫人物等。
搜集证据(简称搜证)：每个推理任务都存在至少一个证据，不同的证据具有不同的剧情推动作用，由玩家对虚拟道具或其他故事角色进行相应操作后获得。
角色信息：不同的人物角色具有不同的角色信息，比如姓名、年龄、性别、外形、性格、成长背景、交际关系和日程安排表等。
公开信息：是推理任务中的全部故事角色(或至少两个故事角色)具有查看权限的角色信息或搜证信息。
私有信息：是推理任务中仅某一个故事角色具有查看权限的角色信息。比如对于第一故事角色的多条信息中，存在信息A只有第二故事角色具有查看权限，那么信息A即为第二故事角色的私有信息。
本申请提供了一种基于AR向用户呈现故事场景的推理任务的互动方案。该推理任务可以是剧本杀、密室逃脱等具有推理环节的游戏任务。本申请可以将故事场景、故事人物、角色信息中的至少一种内容采用AR技术呈现给玩家。
以剧本杀为例，不同的玩家使用不同的剧本和AR终端，AR终端可以是智能手机、平板电脑、电子书阅读器、膝上便携计算机、台式计算机和AR眼镜中的至少一种。例如用户A、用户B、用户C和用户D在真实场景中围坐在一个四人桌前，用户E站在四人桌旁边。在开始玩剧本杀游戏后，用户A、用户B、用户C、用户D和用户E均持有一个AR终端。五个用户中的某个用户(任一用户或主持人用户或管理员用户)，在至少两个候选故事场景中选择一个推理任务。如图1所示，AR终端对实景画面进行图像语义识别后，得到背景区域101和前景人物区域102，前景人物区域102包括脸部区域1021和非脸部区域1022。对脸部区域1021进行脸部识别后，得到实景人物的人物身份并分别与玩家已选择的故事角色绑定。例如用户A绑定第一故事角色，用户B绑定第二故事角色，用户C绑定第三故事角色，用户D绑定第四故事角色，用户E绑定第五故事角色。
图1示出了用户E的摄像头采集到的实景画面100的示意图，该实景画面上包括背景区域101和前景人物区域102，前景人物区域102对应实景人物用户A、用户B、用户C、用户D。示例性的，背景区域101显示为一个房间内部，有矮柜陈设。实景人物区域包括脸部区域1021和非脸部区域1022。用户A侧对用户E的摄像头，脸部区域1021显示其真实样貌与框架眼镜，非脸部区域1022显示其扎丸子头且身穿无袖上衣。用户B面对用户E的摄像头，脸部区域1021显示其真实样貌，非脸部区域1022显示其有八字刘海和及肩长发，且身穿V字领半袖上衣。用户C面对用户E的摄像头，脸部区域1021显示其真实样貌，非脸部区域1022显示其及肩长发且身穿吊带背心。用户D背对用户E的摄像头，非脸部区域1021显示其头发微卷且身穿短袖上衣。
各个用户的AR终端会基于故事场景的场景素材将实景画面100中的背景区域101替换为AR背景区域201，基于故事角色的角色素材对实景人物的非脸部区域1022进行替换，从而将前景人物区域102替换为AR人物区域202，从而将实景画面100替换为AR画面200。在介绍阶段和搜证阶段中的至少一个阶段，AR终端还会在AR画面中显示AR信息控件203。
在推理任务开始后，玩家可以基于AR画面200获取其他故事角色对应的角色信息。角色信息获取方式包括：从服务器获取、语音录入(Optical Character Recognition，OCR)扫描和键盘输入中的至少一种。示例性的，在公聊、私聊或搜证阶段，玩家可以对故事角色、AR道具和AR场景中的至少一项进行操作或互动，与自身对应的故事角色也会做出同样的动作，从而获取故事场景中其他故事角色的搜证信息，并于AR信息控件203中显示该信息。示例性的，基于AR画面200，用户D从服务器中获取已绑定的第四故事角色的基本角色信息：角色四，女，16岁，杨太医之女；公聊阶段，用户A、用户B、用户C和用户E得知角 色四历来与受害者交好，近期却关系恶化；用户E与角色四私聊后得知角色四昨日去过案发场地；用户E对角色四的匣子搜证后发现损坏的银钗。
图2示出了用户E的AR终端显示的AR画面200，该画面上包括AR背景区域201、AR人物区域202和AR信息控件203，AR背景区域201显示推理任务的故事场景，AR人物区域202显示穿戴有AR装扮204的实景人物，AR装扮204对应故事场景中的故事角色，AR信息控件203显示故事角色的搜证信息(基本信息、公开信息和私有信息中的至少一种)，AR信息控件203可位于故事角色的周侧或搜证信息获取的位置。示例性的，AR背景区域201显示为城外山脚的柳树下。AR人物区域112显示穿戴不同古风AR服饰装束的用户A、用户B、用户C和用户D，并且用户B正在弹奏AR虚拟古琴。用户D绑定的第四故事角色的信息显示于其周侧的AR信息控件120上。用户E于私聊阶段获取的信息为私有信息，用户A、用户B、用户C、用户D无法查看，用户E于搜证阶段获取的信息选择对其他玩家公开，用户A、用户B、用户C、用户D均可以查看。
用户A、用户B、用户C、用户D和用户E经过公聊、私聊和搜证后，投票选出用户D绑定的故事角色四为凶手，投票结果正确，推理任务完成，可选择主持人进行复盘。
图3示出了本申请一个示例性实施例提供的计算机系统的结构框图。该计算机系统300包括第一终端310、服务器320和第二终端330。
第一终端310具有摄像头，安装和运行有支持AR互动和推理任务的应用程序。第一终端310是第一用户使用的AR终端。
第一终端310通过无线网络或有线网络与服务器320相连。
服务器320包括一台服务器、多台服务器、云计算平台和虚拟化中心中的一种。示意性的，服务器320包括处理器321和存储器322，存储器322又包括接收模块3221、显示模块3222和控制模块3223。服务器320用于支持AR互动和推理任务的应用程序提供后台服务。可选地，服务器320承担主要计算工作，第一终端310和第二终端330承担次要计算工作；或者，服务器320承担承担次要计算工作，第一终端310和第二终端330承担主要计算工作；或者，服务器320、第一终端310和第二终端330三者之间采用分布式计算架构进行协同计算。
第二终端330具有摄像头，安装和运行有支持AR互动和推理任务的应用程序。第二终端330是第二用户使用的AR终端。
可选地，第一故事角色和第二故事角色处于同一故事场景中。可选地，第一故事角色和第二故事角色可以属于同一个队伍、同一个组织、具有好友关系或具有临时性的通讯权限。
可选地，第一终端310和第二终端330上安装的应用程序是相同的，或两个终端上安装的应用程序是不同控制系统平台的同一类型应用程序。第一终端310可以泛指多个终端中的一个，第二终端330可以泛指多个终端中的一个，本实施例仅以第一终端310和第二终端330来举例说明。第一终端310和第二终端330的设备类型相同或不同，该设备类型包括：智能手机、平板电脑、电子书阅读器、膝上便携计算机、台式计算机和AR眼镜中的至少一种。以下实施例以终端包括手机和AR眼镜来举例说明。
本领域技术人员可以知晓，上述终端或故事角色的数量可以更多或更少。比如上述终端或故事角色可以仅为一个，或者上述终端或故事角色为几十个或几百个，或者更多数量。本申请实施例对终端或故事角色的数量和设备类型不加以限定。
需要说明的是，本申请所涉及的信息(包括但不限于用户设备信息、用户个人信息等)、数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)以及信号，均为经用户授权或者经过各方充分授权的，且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。例如，本申请中涉及到的实景视频流都是在充分授权的情况下获取的，终端和服务器仅在程序运行期间缓存该实景视频流，并不会固化存储和二次利用该实景视频流的相关数据。
图4示出了本申请一个示例性实施例提供的基于故事场景的人机互动方法的流程图。本实施例以该方法由图3所示的终端执行来举例说明。该终端具有摄像头。该方法包括：
步骤220：获取摄像头采集的实景视频流，实景视频流的视频画面包括背景区域和前景人物区域；
终端内安装和运行有支持AR互动和推理任务的应用程序，该推理任务可以是剧本杀或密室逃脱等推理游戏中的至少一种。
以剧本杀为例，终端接收到用户启动该应用程序的操作后，显示至少两个候选故事场景，终端接收到用户选定任一故事场景的操作后，显示至少两个候选故事角色，终端接收到用户选定任一故事角色的操作后，还将用户的脸部数据与选定的该故事角色进行绑定。
以密室逃脱为例，终端接收到用户启动该应用程序的操作后，显示至少两个候选故事场景，终端接收到用户选定任一故事场景的操作后，显示至少两个候选故事角色，终端接收到用户选定任一故事角色的操作后，还将用户的脸部数据与选定的该故事角色进行绑定。
终端获取摄像头采集的实景视频流。实景视频流包括多个实景视频帧，每个实景视频帧构成一个实景视频画面。多个实景视频帧按照时序排列显示成连续的视频画面。在本实施例中，通过图像语义识别将实景视频帧分割成背景区域和前景人物区域，多个实景视频帧按时序排列显示为实景视频流。该实景视频流的全部或部分实景视频帧包括背景区域和前景人物区域。
背景区域是指终端的摄像头采集到的实景视频帧中，衬托实景人物的景物或布景区域。比如背景区域是房间的墙壁和家具。
前景人物区域是指终端的摄像头采集到的实景人物区域。实景人物区域包括脸部区域和非脸部区域。脸部区域是指摄像头采集到的实景画面中实景人物的脸部区域，非脸部区域是指摄像头采集到的实景画面中，实景人物除脸部区域以外的区域，比如躯干区域和四肢区域。
图像语义识别：是指计算机对图像进行处理、分析和理解，以识别同一帧视频帧或图像中，不同语义对象所在的二维区域的技术。比如，在同一帧实景视频帧中，区分出背景区域、脸部区域和非脸部区域。
步骤240：基于实景视频流显示AR视频流；
AR视频流包括多个AR视频帧，每个AR视频帧构成一个AR视频画面，多个AR视频帧按照时序排列显示成连续的AR视频画面。在一些实施例中，AR视频流中的AR视频帧与实景视频帧存在一一对应关系。在一些实施例中，AR视频流中属于关键帧的AR视频帧与实景视频流中属于关键帧的实景视频帧存在一一对应关系。
在本实施例中，AR视频帧包括AR背景区域和AR人物区域，AR人物区域显示穿戴有AR装扮的实景人物，AR装扮对应故事场景中的故事角色。AR背景区域是对背景区域中的画面内容处理得到的，AR人物区域是对前景人物区域中的画面内容处理得到的。
AR背景区域：是指应用程序运行时显示的虚拟背景，该AR背景区域是基于故事场景的场景素材对实景视频帧中的背景区域进行替换，将虚拟内容和真实存在的内容进行实时融合，形成虚拟与现实之间互动的三维环境。
AR人物区域：是指应用程序运行时显示的穿戴有AR装扮的实景人物区域，该AR人物区域是基于故事角色的角色素材对实景人物的非脸部区域进行替换，将虚拟内容和实景人物的脸部区域进行实时融合，形成的虚拟与现实之间互动的三维人物区域。
在一个推理任务中存在至少一个故事场景。一个故事场景中存在至少一个故事角色。每个故事角色具有至少一组AR装扮。可选地，一个故事角色在不同的故事场景中有不同的AR装扮，一个故事角色在同一个故事场景的不同时间段对应有不同的AR装扮，或者，一个故事角色在同一个故事场景的不同地点对应有不同的AR装扮。
步骤260：响应于交互操作，改变AR视频流的显示内容。
在一些实施例中，响应于与AR背景区域中虚拟物品的物品交互操作，改变AR背景区 域的显示内容。
可选地，物品交互操作包括物品触摸操作、物品抓举操作、物品使用操作、物品检查操作、手势指认操作、眼球锁定操作、眼球滑动操作中的至少一种。物品触摸操作指触摸虚拟物品的操作。物品抓举操作指抓举虚拟物品的操作。物品使用操作指使用虚拟物品的操作。物品检查操作指检查虚拟物品的操作。
可选地，改变AR背景区域的显示内容的方式包括如下之一：
(1)响应于与AR背景区域中虚拟物品的物品交互操作，在AR背景区域中显示故事线索。
在一个实施例中，以密室逃脱为例，玩家在触摸AR背景区域中的花瓶后，在AR背景区域中以文字的形式显示逃出密室的方式。
(2)响应于与AR背景区域中虚拟物品的物品交互操作，更新显示AR背景区域中的虚拟物品。
在一个实施例中，以剧本杀为例，玩家B持有组合件1，玩家A将组合件2交给玩家B，显示组合后的组合件1和组合件2。
(3)响应于与AR背景区域中虚拟物品的物品交互操作，更新AR背景区域中的故事场景的场景画面。
在一个实施例中，以密室逃脱为例，在密室场景中，玩家与AR背景区域中的虚拟门交互后，将AR背景区域显示的密室场景更新为户外场景。
需要说明的是，上述不同实施例中的物品交互操作可以是相同的操作，也可以是不同的操作。
在一些实施例中，响应于与AR人物区域中实景人物的人物交互操作，改变实景人物的角色信息。可选地，人物交互操作包括人物触摸操作、人物抓握操作、人物交谈操作、手势指认操作、眼球锁定操作、眼球滑动操作中的至少一种。
可选地，角色信息包括性别、年龄、身份、职业、特长、能力、技能、身高、体重、衣物穿搭中的至少一种。
可选地，改变实景人物的角色信息包括如下之一：
(1)响应于与AR人物区域中故事角色的人物交互操作，将故事角色的第一角色信息更改为第二角色信息。
在一个实施例中，角色信息包括职业，用户A同用户B交谈后，得知用户B的职业是“医生”而非“护士”，故将用户B的职业由“护士”更改为“医生”。
(2)响应于与AR人物区域中故事角色的人物交互操作，新增故事角色的第三角色信息。
在一个实施例中，故事角色A向故事角色B提问，故事角色B回答了故事角色B的职业，新增故事角色B的职业。
(3)响应于与AR人物区域中故事角色的人物交互操作，删除故事角色的第四角色信息。
在一个实施例中，故事角色A需要与故事角色B战斗，在故事角色A战胜故事角色B的情况下，删除故事角色B的角色信息。
(4)响应于与AR人物区域中故事角色的人物交互操作，增加故事角色与其它故事角色之间的角色关系信息。
在一个实施例中，故事角色A同故事角色B交谈后，故事角色A得知故事角色B与故事角色C之间是兄弟关系，则新增故事角色B与故事角色C之间的角色关系信息。
需要说明的是，上述不同实施例中的人物交互操作可以是相同的操作，也可以是不同的操作。
在一些实施例中，响应于故事场景的场景切换操作，改变故事场景的场景画面。
可选地，场景切换操作包括物品触摸操作、物品抓举操作、人物触摸操作、人物抓握操作、手势指认操作、眼球锁定操作、眼球滑动操作中的至少一种。
可选地，响应于故事场景的场景切换操作，将故事场景的第一场景画面切换为第二场景画面，第一场景画面和第二场景画面是不同的场景画面。
在一些实施例中，响应于故事场景的故事情节触发操作，改变AR视频流中与故事情节有关的显示内容。
可选地，故事情节触发操作包括故事情节发展到预设时间点、故事角色触发的推进故事情节发展的操作、主持人触发的推进故事情节发展的操作中的至少一种。
可选地，响应于故事场景的故事情节触发操作，改变AR视频流中与故事情节有关的显示内容的方式包括如下之一：
可选地，场景切换操作包括物品触摸操作、物品抓举操作、人物触摸操作、人物抓握操作、手势指认操作、眼球锁定操作、眼球滑动操作中的至少一种。
(1)响应于故事场景的故事情节触发操作，改变故事场景的场景画面。
在一些实施例中，故事角色注视虚拟物品达到8秒后，将故事场景从古风仙侠场景切换为现代都市场景。
(2)响应于故事场景的故事情节触发操作，在AR背景区域中新增情节道具。
在一些实施例中，故事角色做出跳跃动作后，在AR背景区域中显示新增的情节道具，该情节道具用于推动故事情节的发展。
(3)响应于故事场景的故事情节触发操作，更新故事角色的角色信息。
在一些实施例中，故事角色A向故事角色B说出故事角色A的职业后，更新故事角色A的职业。
需要说明的是，上述不同实施例中的故事情节触发操作可以是相同的操作，也可以是不同的操作。
步骤280：基于改变后的显示内容，完成故事场景对应的推理任务。
基于AR视频流中显示的AR信息，完成故事场景对应的信息获取任务、证据搜集任务和谜题推理任务中的至少一种。该AR信息包括关联在AR角色的周侧显示的AR信息、关联在虚拟道具的周侧显示的AR信息、关联在虚拟环境中显示的AR信息、关联在非玩家角色(Non-Player Character，NPC)上显示的AR信息中的至少一种。
该AR信息包括与完成推理任务有关的文字信息、图片信息、视频信息、音频信息、动画信息、特效信息中的至少一种。
信息获取任务是用于获取各个故事角色的角色信息的任务。
证据搜集任务是用于搜集推理任务的相关证据信息的任务。
谜题推理任务是基于已经获取的角色信息和/或相关证据信息进行谜题推理的任务。
以推理任务是剧本杀游戏为例，该推理任务包括：信息获取任务、证据搜集任务和谜题推理任务中的至少一种。示意性的，信息获取任务包括介绍阶段、公聊阶段、私聊阶段、搜证阶段、结案阶段中的至少一个阶段。证据搜集任务包括搜证阶段，谜题推理任务包括结案阶段。
在一个实施例中，玩家在剧本介绍阶段获取故事背景信息，在公聊阶段获取故事角色的基本信息，在私聊阶段获取故事角色的私有扩展信息，在搜证阶段对虚拟场景、虚拟道具或NPC进行搜证操作，得到搜证信息，经过分析和投票后得到推理结果，由终端显示或由主持人宣布结果正确与否，完成推理任务。
以推理任务是密室逃脱为例，该推理任务包括：信息获取任务、证据搜集任务和谜题推理任务中的至少一种。示意性的，信息获取任务包括信息寻找阶段，证据搜集任务包括搜证阶段，谜题推理任务包括逃脱阶段。信息获取阶段、搜证阶段和逃脱阶段。在信息获取阶段，玩家可以通过终端、服务器或工作人员知晓本次密室逃脱的故事背景、推理任务或逃脱目标。
在一个实施例中，玩家通过阅读终端显示的虚拟信息控件得知密室逃脱的故事背景和逃脱目标，在搜证阶段对虚拟场景、虚拟道具或NPC进行搜证操作后，得到逃离密室的方法，成功逃离密室后，完成推理任务。
可选地，基于AR视频流中显示的AR信息，完成故事场景对应的互动推理任务。其中，互动推理任务是与故事场景的场景画面互动的任务，或者，互动推理任务是与故事场景中的故事角色互动的任务。
在一个实施例中，玩家通过阅读终端显示的虚拟信息控件得知逃离密室的方法是完成故事场景内的拼图，在搜证阶段将拼图碎片放入指定位置以完成拼图，达成逃离密室的条件，完成拼图后，完成互动推理任务。
在一个实施例中，玩家得知进入剧本杀下一阶段的任务是获取故事角色持有的钥匙，在搜证阶段与故事角色互动后，获取故事角色持有的钥匙，进行剧本杀的下一阶段，获取钥匙后，完成互动推理任务。
综上所述，本实施例提供的方法，通过将实景视频流中的背景区域替换为推理任务对应AR背景区域，将实景视频流中的实景人物替换为穿戴有AR装扮的实景人物，从而实现采用AR场景和AR装扮来营造剧本杀或密室逃脱的故事场景，无需借助专门定制的服装、道具和纸件剧本，就能够提供出非常有沉浸感的视觉效果。在减少专门定制的物理资源的耗费的情况下，实现比相关技术更加优秀的视觉体验。
在一个可能的实施例中，上述步骤240包括步骤241、步骤243、步骤245、步骤247和步骤249。
图5示出了本申请一个示例性实施例提供的基于实景视频流显示AR视频流的方法的流程图。本实施例以该方法由图3所示的终端和/或服务器执行来举例说明。该方法包括：
步骤241：对实景视频流中的视频帧进行图像语义识别，得到背景区域和前景人物区域，前景人物区域包括脸部区域和非脸部区域；
示例性的，用户A、用户B、用户C和用户D在真实场景中围坐在一个四人桌前，用户E站在四人桌旁边，各个用户使用的AR终端的摄像头采集到不同的实景视频流，该实景视频流由各个终端分别上传至服务器中，实景视频流包含多个实景视频帧，服务器对每个实景视频帧进行图像语义识别，得到每个实景视频帧中的背景区域和前景人物区域。
以用户E的终端上传的实景视频流为例，服务器可以在分析后识别出如图1所示的实景视频流，背景区域显示房间内部和矮柜，前景人物区域显示用户A、用户B、用户C和用户D的脸部、躯干与四肢的真实情况。
步骤243：对脸部区域进行脸部识别，得到实景人物的人物身份；
由服务器识别出的各个实景人物的脸部区域分别对应一套脸部数据，服务器对脸部数据进行身份识别后，可以确定每个实景人物的人物身份。
示例性的，用户A、用户B、用户C、用户D和用户E的脸部区域均在服务器中分别对应一套脸部数据，用户E持有的AR终端的摄像头对这四名用户中的任意一位进行脸部采集，均可以得到其对应的实景人物身份。
步骤245：基于实景人物的人物身份，确定实景人物在推理任务中绑定的第一故事角色；
在一个示例性的实施例中，在推理任务开启前，用户A、用户B、用户C、用户D和用户E各自使用AR终端选择故事角色，由服务器将其脸部数据与故事角色进行绑定。
在一个示例性的实施例中，在推理任务开启前，由五个用户中的某个用户(任一用户或主持人用户或管理员用户)选择或分配故事角色，由服务器将每个用户的脸部数据与其对应的故事角色进行绑定。
服务器存储各个用户的脸部数据于故事角色之间的绑定关系。在推理任务开启后，当识别出实景视频帧中出现的实景人物的人物身份后，基于该绑定关系确定实景人物在推理任务中绑定的第一故事角色。
可选地，对于其它人物身份，基于该绑定关系确定实景人物在推理任务中绑定的第二故事角色、第三故事角色等等。
步骤247：基于故事场景的场景素材对背景区域进行替换，得到AR背景区域；基于第一故事角色的角色素材对非脸部区域进行替换，得到AR人物区域，得到AR视频流；
服务器确定故事场景的场景素材，以及与各个故事角色对应的角色素材。
在一个实施例中，服务器分别对用户A、用户B、用户C、用户D和用户E采集到的实景视频流中的背景区域中的内容进行计算，将故事场景的场景素材渲染至背景区域，得到AR背景区域；服务器对每个用户对应的前景人物区域的非脸部区域中的内容进行计算，将其已绑定的故事角色的角色素材渲染至前景人物区域，得到穿戴有AR装扮的实景人物，得到AR人物区域。示例性的，服务器将故事场景的场景素材分别渲染至用户A、用户B、用户C、用户D和用户E采集到的实景视频流的背景区域中，以用户A为例，服务器将用户A绑定的第一故事角色的角色素材渲染至其他用户采集到的用户A的非脸部区域，得到AR视频流。
在一个实施例中，服务器对用户E采集到的实景视频流中的背景区域中的内容进行计算，将故事场景的场景素材渲染至背景区域，得到AR背景区域；服务器320对每个用户对应的前景人物区域的非脸部区域中的内容进行计算，将其已绑定的故事角色的角色素材渲染至前景人物区域，得到穿戴有AR装扮的实景人物，得到AR人物区域。示例性的，服务器将故事场景的场景素材渲染至用户E采集到的实景视频流中，并将每位用户绑定的故事角色的角色素材分别渲染至用户E采集到的非脸部区域，得到AR视频流。
可选地，故事角色的AR装扮可以由本地创建并上传至服务器中，或由本地自定义并上传至服务器中。
示例性的，本地可以对故事角色的AR装扮进行创建，例如建模等，将创建的AR装扮上传至服务器中；或者，自定义AR装扮，例如对已有的AR装扮进行调整，如调整大小、形状、间距、换色等，将自定义的AR装扮上传至服务器中。
步骤249：基于AR背景区域和AR人物区域，显示AR视频流。
在一个实施例中，各个用户的终端显示的AR视频流，分别由各自的终端的摄像头采集实景视频流后，上传至服务器进行处理后，由服务器再传回其终端。以用户A为例，其使用的终端上传实景视频流，经服务器处理后得到AR背景区域和AR人物区域，AR人物区域对应穿戴有AR装扮的用户B、用户C、用户D和用户E。
在一个实施例中，各个用户的终端显示的AR视频流，由其中任意一个终端的摄像头采集实景视频流后，上传至服务器进行处理后，由服务器再传回所有终端。示例性的，用户E使用的终端上传实景视频流，经处理后得到AR背景区域和AR人物区域，AR人物区域对应穿戴有AR装扮的用户A、用户B、用户C、用户D和用户E，每个终端选择性显示其对应视角的AR视频流。
上述过程可以由计算机系统在一次推理任务游戏中执行多次。在终端的计算能力较强的情况下，上述过程也可以均由终端来执行，无需服务器的配合。
综上所述，本实施例提供的方法，通过图像语义分割来识别实景视频流中的背景区域和前景人物区域，前景人物区域包括脸部区域和非脸部区域。
在一些实施例中，AR终端是AR眼镜。在实景视频流中显示的实景人物佩戴有AR眼镜，由于AR眼镜与故事场景可能是不匹配的，比如故事场景是古风仙侠场景时，出现作为现代化消费电子设备的AR眼镜不符合古风视觉场景，因此本申请实施例提供了一种基于故事场景的人机互动方法中的AR视频流中不显示实景人物佩戴的AR眼镜的方法。
图6示出了本申请实施例提供的AR视频流中不显示实景人物穿戴AR眼镜的方法的示意图。示例性的，服务器将终端采集并上传的实景人物的样本人脸数据与AR视频流中显示的第一脸部画面601输入至生成式网络进行图像重建，得到实景人物未佩戴AR终端的第二 脸部画面603，在AR视频流中的AR人物区域的第二脸部区域中，显示该第二脸部画面603。
在一个示例中，生成式网络是具有图像重建能力的神经网络。该生成式网络包括判别器和生成器。在训练过程中，需要判别器和生成器的配合训练；在应用过程中，仅需要生成器即可。
在训练过程中，训练集中包括不同用户的多组样本数据，每组数据包括同一用户的样本脸部画面(佩戴有AR终端)和样本人脸数据(未佩戴有AR终端，比如用户的正脸图像)。计算机设备将同一用户的样本脸部画面和样本人脸数据输入至生成器，由生成器重建出预测脸部画面。其中，样本脸部画面和预测脸部画面的人脸角度(可能是任意角度)相同，但预测脸部画面未佩戴有AR终端。可选地，样本人脸数据是未佩戴有AR终端的正脸图像，用于模拟用户在绑定阶段采集到的人脸。样本脸部画面的人脸角度与样本人脸数据的人脸角度可以是不同的。
判别器用于对预测脸部画面或样本脸部画面进行判别，识别是否为生成器重建的图像或原始图像。基于生成式网络的交替训练方式，固定判别器的网络参数，对生成器的网络参数进行更新；或者，固定生成器的网络参数，对判别器的网络参数进行更新。直至误差收敛或训练次数达到预设次数，得到训练完毕的判别器和生成器。
在应用阶段，计算机设备将终端采集并上传的实景人物的样本人脸数据与AR视频流中显示的第一脸部画面输入至生成器进行图像重建，得到实景人物未佩戴AR终端的第二脸部画面。
图7示出了本申请另一个示例性实施例提供的基于故事场景的人机互动的方法的流程图。本实施例以该方法由图3所示的终端和/或服务器执行，以如图1所示的用户A、用户B、用户C、用户D和用户E各持有一个终端进行同一推理任务来举例说明。该终端具有摄像头。该方法包括：
步骤211：显示至少两个候选故事场景的任务选择控件；
任务选择控件是用于在至少两个候选故事场景中选择一个故事场景的控件。任务选择控件可显示为下拉控件、卡片控件或勾选控件等形式。
终端接收到用户开启推理任务的开启操作后，显示至少两个候选推理任务的任务选择控件，每个推理任务对应一个故事场景。
以终端为手机为例，用户可以对应用程序进行触摸操作(包括但不限于点击、双击和滑动等)开启推理任务，应用程序界面显示至少两个候选推理任务，如谍战类任务、西部牛仔类任务、古风玄幻类任务和古墓探险类任务，每个推理任务的封面都对其故事场景有简单介绍，用户可以通过滑动手机界面查看。
以终端为AR眼镜为例，用户可以对悬浮在空中或平铺在面前的桌子上的虚拟选择控件进行触摸操作(包括但不限于点击、双击、拉拽、拖动和滑动等)来开启推理任务，虚拟选择控件上显示至少两个候选推理任务，如谍战类任务、西部牛仔类任务、古风玄幻类任务和古墓探险类任务，每个推理任务的封面都对其故事场景有简单介绍，用户可以通过滑动操作或翻阅操作查看。
步骤212：响应于任务选择控件上的选择操作，确定至少两个候选故事场景中被选择的故事场景；
选择操作是用于对任务选择控件中显示的故事场景进行选择的操作。选择操作可以是在下拉控件中滑动选取，在卡片控件中拖拽选取，或在勾选控件中点击选取等形式。
以终端为手机为例，由用户E在其持有的手机界面上点击推理任务“古风故事一”的选择控件，用户A、用户B、用户C、用户D和用户E的手机界面均显示进入“古风故事一”的故事场景。
以终端为AR眼镜为例，虚拟选择控件平铺在面前桌上，5位用户均可以滑动查看各个推理任务，用户E将“古风故事一”选择控件拉拽到任一用户的身上，用户A、用户B、用 户C、用户D和用户E的AR眼镜均显示进入“古风故事一”的故事场景。
步骤213：显示故事场景中的至少两个候选故事角色的角色选择控件；
角色选择控件是用于在至少两个候选故事角色中选择一个故事角色的控件。角色选择控件可显示为下拉控件、卡片控件或勾选控件等形式。
以终端为手机为例，手机界面上显示至少5个候选故事角色，例如：角色一、角色二、角色四、角色三和角色五。
以终端为AR眼镜为例，虚拟选择控件悬浮在空中，显示至少5个候选故事角色的虚拟选择控件，例如：角色一、角色二、角色四、角色三和角色五。
步骤214：响应于角色选择控件上的选择操作，确定至少两个候选故事角色中被选择的故事角色；
以终端为手机为例，由每位用户在其持有的手机界面上点击故事角色选择控件来选择故事角色，例如，用户A点击角色一的选择控件、用户B点击角色二的选择控件、用户C点击角色三的选择控件、用户D点击角色四的选择控件、用户E点击角色五的选择控件，完成故事角色选择。
以终端为AR眼镜为例，用户A将角色一的虚拟选择控件拖动到自己身上来选择角色一，用户E将角色二、角色三、角色四和角色五的选择控件分别拉拽到用户B、用户C、用户D和自己身上，完成故事角色选择。
步骤215：将被选择的故事角色与终端对应的实景人物的脸部数据进行绑定；
以终端为手机为例，每位用户用持有的终端的摄像头采集自己的脸部数据。
在一个实施例中，手机将采集到的脸部数据的上传至服务器，服务器将该脸部数据与用户已选择的故事角色绑定，例如，用户A的脸部数据与角色一绑定，用户B的脸部数据与角色二绑定，用户C的脸部数据与角色三绑定，用户D的脸部数据与角色四绑定，用户E的脸部数据与角色五绑定。
在一个实施例中，手机对采集到的脸部数据进行计算，手机将该脸部数据与用户已选择的故事角色绑定，例如，用户A的脸部数据与角色一绑定，用户B的脸部数据与角色二绑定，用户C的脸部数据与角色三绑定，用户D的脸部数据与角色四绑定，用户E的脸部数据与角色五绑定。
以终端为AR眼镜为例，以用户E来举例说明，用户E持有的AR眼镜采集到用户A、用户B、用户C和用户D的脸部数据。
在一个实施例中，用户E的AR眼镜将这四位用户的脸部数据上传至服务器，由服务器将用户A、用户B、用户C和用户D的脸部数据分别与其已选择的角色一、角色二、角色三和角色四进行绑定，用户A持有的AR眼镜采集到用户E的脸部数据后上传至服务器，服务器将用户E的脸部数据与其已选择的角色五进行绑定。
在一个实施例中，用户E的AR眼镜对这四位用户的脸部数据进行计算，将用户A、用户B、用户C和用户D的脸部数据分别与其已选择的角色一、角色二、角色三和角色四进行绑定，用户A持有的AR眼镜采集到用户E的脸部数据后进行计算，得到用户E的脸部数据，将该脸部数据与其已选择的角色五进行绑定。
在一些实施例中，如果推理任务过程中突然有5位用户以外的人员进入该房间，服务器会将由任一用户持有的终端的摄像头采集到的该人员的脸部数据，与该故事场景中的对剧情无推动作用的NPC进行绑定。示例性的，保洁人员突然进入该房间，用户E的摄像头采集到该保洁人员的脸部数据并由AR终端上传至服务器，服务器将其脸部数据与该故事场景中的NPC扫地丫鬟绑定，用户A、用户B、用户C、用户D和用户E的终端显示该保洁人员为穿戴有古风AR装扮的正在扫地的丫鬟。
步骤220：获取摄像头采集的实景视频流，实景视频流画面包括背景区域和前景人物区域；
以用户E为例，其持有的AR终端的摄像头采集到如图1所示的实景视频流。
在一个实施例中，该实景视频流由终端上传至服务器后，服务器经过图像语义识别，识别出背景区域和前景人物区域，背景区域为一个房间内，有矮柜陈设，前景人物区域为显示真实样貌的用户A、用户B、用户C和用户D。
在一个实施例中，摄像头采集到实景视频流后，终端对该实景视频流进行图像语义识别，识别出背景区域和前景人物区域，背景区域为一个房间内，有矮柜陈设，前景人物区域为显示真实样貌的用户A、用户B、用户C和用户D。
步骤240：基于实景视频流显示AR视频流，AR视频流画面包括AR背景区域和AR人物区域；
在一个实施例中，每个用户的终端将该故事场景的场景素材和角色素材替换至其摄像头采集到的实景视频流中，可选地，该场景素材或角色素材可以是终端从服务器获取的，或者，该场景素材或角色素材是从终端上读取的。
在一个实施例中，服务器将该故事场景的场景素材和角色素材替换至用户E上传的实景视频流中，再将替换得到的AR视频流传回5位用户的终端，每个终端依照各自的视角显示相应的AR视频流。
可选地，该场景素材是基于用户所处的真实环境的时间和空间布置而形成的虚拟内容与真实存在的内容实时融合的三维场景；或者，该场景素材是与用户所处的真实环境的时间与空间不一致的三维场景。
例如，用户A、用户B、用户C、用户D和用户E持有的终端均可以从服务器获取“古风故事一”的场景素材(如城外山脚、福聚楼，角色一、角色二、角色四、角色三和角色五的寝居等)和角色素材(如角色一、角色二、角色四、角色三和角色五的服饰装扮等)，再由各自的终端将场景素材和角色素材替换至采集到的实景视频流中。公聊阶段五位用户均位于城外山脚这一场景，该场景具有的山、水、树木、天空和地形等AR素材与用户真实所处的房间的三维结构不一致，营造出了更为广阔的户外视觉效果。搜证阶段五位故事角色的寝居依照用户真实所处的房间布置，如：虚拟的墙壁渲染至真实的墙壁上，虚拟的床渲染至真实的角落，虚拟的桌柜渲染至真实的桌柜上等。
在基于AR视频流完成故事场景对应的推理任务的过程中，推理任务包括：信息获取任务、搜集证据任务和谜题推理任务中的至少一种。
针对信息获取任务：
步骤262：获取第一故事角色的角色信息；
终端可以从服务器获取、语音录入、OCR扫描和键盘输入中的至少一种方式获取第一故事的角色信息。
在一个实施例中，服务器存储有所有故事角色的角色信息，推理任务进入特定阶段后，终端可以自动从服务器获取相应的角色信息。
在一个实施例中，用户在获取当前故事角色或其他故事角色的角色信息后，可通过语音将该角色信息录入终端。比如，用户持有纸件剧本，通过读出纸件剧本上的文字内容，通过语音将纸件剧本上的故事角色的角色信息录入终端。
在一个实施例中，用户可以利用OCR扫描含有故事角色信息的图片、纸件剧本、虚拟纸质道具等，获取相应信息。
在一个实施例中，用户可以通过键盘输入将故事角色的角色信息录入至终端，该键盘可以是显示于智能手机、平板电脑、便携机算机、电子书阅读器等具有摄像头的终端界面上的键盘，也可以是由AR眼镜显示的虚拟键盘。
示例性的，公聊阶段，用户E朗读从服务器获取的角色五的身份信息和人际关系简介，用户A和用户C通过语音录入用户E的声音获取角色五的身份信息和人际关系简介，用户B通过对桌面上的信息图片进行OCR扫描获取角色五的身份信息和人际关系简介，用户D通 过手机键盘或AR虚拟键盘输入角色五的身份信息和人际关系简介。
示例性的，用户E与用户D私聊，得知用户D绑定的角色四在案发前一天去过福聚楼的案发现场，将该信息通过键盘输入至AR终端。
步骤264：在AR视频流中显示第一AR信息，第一AR信息用于将角色信息关联至第一故事角色对应的实景人物进行显示；
在一个实施例中，用户B的AR终端上显示有位于用户A周侧的第一AR信息控件，用户B获取的有关于角色一的角色信息显示于该第一AR信息控件上。用户B在公聊阶段获取的关于角色一的公开信息，用户C、D和E也可以在自己持有的终端上显示的位于用户A周侧的第一AR信息控件上查看到；用户B在私聊阶段获取的关于角色一的私有信息，不显示于用户C、D和E各自持有的终端上显示的位于用户A周侧的第一AR信息控件上。
示例性的，用户E的周侧显示关于角色五的第一AR信息控件，其余用户在推理任务过程中获取的与角色五有关的信息会显示在该第一AR信息控件上。
示例性的，用户E于私聊阶段获取的角色四在案发前一天去过案发现场这一信息，显示在位于用户D周侧的第一AR信息控件上，且由于该信息是用户E的私有信息，用户A、B、C和D无法在用户D周侧的第一AR信息控件上看到该信息。
示例性的，已获取的角色信息可以按照用户获取时间或者推理任务中的时间线进行排序，便于用户进行查看、分析和推理。
示例性的，该信息控件中的信息可以是以文字描述、图片表述、语音描述或视频播放等形式中的至少一种进行显示的。
综上所述，本实施例提供的方法，通过显示至少两个故事场景的推理任务和至少两个故事场景的故事角色，并接收用户的选择操作，提供更丰富的游戏体验；将故事角色与用户的脸部数据进行绑定，将实景视频流替换为AR视频流，提供更沉浸的视觉体验和代入感；通过接收第一故事角色的角色信息和搜证信息的获取操作，并于AR视频流中显示角色信息和搜证信息，减少信息记录成本、提升信息查看的趣味性，也方便用户将故事角色与信息进行绑定。
针对证据搜集任务：
步骤266：获取第一故事角色的搜证信息；
在一个实施例中，用户可以对该故事场景中的虚拟场景进行搜证操作，该操作包括但不限于触摸、点击、放大、缩小和拼接等，获取虚拟场景中的搜证信息。
在一个实施例中，用户可以对该故事场景中的虚拟道具进行搜证操作，该操作包括但不限于打开、关闭、摔碎、拼接、调整位置和敲击等，获取虚拟道具上的搜证信息。
在一个实施例中，用户可以对该故事场景中的NPC进行搜证操作，该操作包括但不限于攻击、躲避、触摸、拥抱和交流等，获取NPC身上的搜证信息。
步骤268：在AR视频流中显示第二AR信息，该第二AR信息用于显示第一故事角色的搜证信息。
在一个实施例中，用户A在虚拟场景中获取的搜证信息显示于该虚拟场景内的第二AR信息控件上，若用户A选择公开，则全部故事角色都有查看权限，若用户A选择私有，则其他故事角色均无法查看。
在一个实施例中，用户B在虚拟道具上获取的搜证信息显示于该虚拟道具周侧的第二AR信息控件上，若用户B选择公开，则全部故事角色都有查看权限，若用户B选择私有，则其他故事角色均无法查看。
在一个实施例中，用户C在搜证阶段获取的某一故事角色的搜证信息显示于该用户周侧的第二AR信息控件上，若用户C选择公开，则全部故事角色都有查看权限，若用户C选择私有，则其他故事角色均无法查看。
以上基于故事场景获取第一故事角色的搜证信息的方法可实现为以下三种情况：
第一种情况(基于虚拟场景进行搜证)，如图8所示：
步骤266a：在搜证阶段，显示与第一故事角色相关的虚拟场景；
示例性的，进入搜证阶段后，用户可以自由选择要进行搜证的虚拟场景，可选地，该虚拟场景中无NPC，或者，该虚拟场景中存在NPC。
例如，用户E想要对角色四的寝居进行搜证，在AR终端显示的AR虚拟场景中选择角色四的寝居，该选择可以通过点击、滑动、拉拽、视线固定超过5秒等方式实现。
步骤266b：响应于对虚拟场景的搜证操作，获取第一故事角色在虚拟场景中的第一搜证信息；
示例性的，用户C在角色四寝居内进行搜证，发现家具摆设杂乱，像是因翻找什么东西而弄乱的，用户C的终端对准杂乱的家具3秒后，终端识别并获取到该搜证信息。
步骤268a：在AR视频流中显示位于第一故事角色对应的实景人物周侧的第二AR信息控件，第二AR信息控件显示有第一故事角色的搜证信息。
在一个实施例中，用户C发现用户D绑定的角色四的寝居内家具因翻找东西而杂乱，这一搜证信息显示于用户D周侧的AR信息控件上。
该搜证信息可选择公开或私有，若选择公开，则该故事场景中的其他故事角色也可查看；若选择私有，则该故事场景中的其他故事角色无法查看。
第二种情况(基于虚拟道具进行搜证)，如图9所示：
步骤266c：在搜证阶段，显示与第一故事角色相关的虚拟道具；
示例性的，进入搜证阶段后，用户的AR终端可以显示出与某一故事角色相关的虚拟道具，可选地，该虚拟道具存在于特定虚拟场景中，或者，该虚拟道具无需存在于特定场景中。
例如，用户E选择对用户B的绑定的角色二的古琴进行搜证，用户E在“古风故事一”推理任务的任一场景下均可以选择查看该古琴。
例如，用户E选择对用户D绑定的角色四的妆奁进行搜证，用户E的终端在用户E进入角色四的寝居后才会显示出该妆奁。
步骤266d：响应于对虚拟道具的搜证操作，获取第一故事角色在虚拟道具上关联的第二搜证信息；
示例性的，用户E对用户B绑定的角色二的AR虚拟古琴进行观察，在AR虚拟古琴上发现一处血迹，用户E将手指在该血迹处放置3秒后，AR终端识别并获取该关于角色二的搜证信息。
示例性的，用户E在用户D绑定的角色四的AR虚拟寝居内进行搜证，其使用AR终端对准角色四的妆奁5秒后，妆奁打开，在妆奁内发现损坏的银钗，用户E使用AR终端对准该损坏的银钗3秒后，AR终端识别并获取该关于角色四的搜证信息。
步骤268b：在AR视频流中，在搜证信息的获取位置显示第二AR信息控件，第二AR信息控件显示有第一故事角色的搜证信息。
在一个实施例中，用户E在用户B绑定的角色二的古琴上发现血迹，这一搜证信息显示于古琴周侧的AR信息控件上。
在一个实施例中，用户B在角色四寝居的妆奁内发现损坏的银钗，这一搜证信息显示于妆奁周侧的AR信息控件上。
该搜证信息可选择公开或私有，若选择公开，则该故事场景中的其他故事角色也可查看；若选择私有，则该故事场景中的其他故事角色无法查看。
第三种情况(基于NPC进行搜证)，如图10所示：
步骤266e：在搜证阶段，显示与第一故事角色相关的NPC；
示例性的，进入搜证阶段后，用户的AR终端可以显示出与某一故事角色相关的NPC，该NPC存在于特定虚拟场景中，或者，该NPC无需于特定存在于特定虚拟场景中。
例如，用户B想要对角色三的寝居内的婢女进行搜证，在AR终端显示的AR虚拟场景 中选择角色三的寝居，该选择可以通过点击、滑动、拉拽、视线固定超过5秒等方式实现，寝居内显示有婢女。
步骤266f：响应于对NPC故事角色的互动操作，获取第一故事角色在NPC故事角色上关联的第三搜证信息；
示例性的，用户B在用户C绑定的角色三的寝居内的婢女进行搜证，将婢女衣袖拉动后发现青紫的伤痕，用户B的终端从服务器获取到“角色三常常打婢女”这一搜证信息。
步骤268c：在AR视频流中，在NPC所处的获取位置显示第二AR信息控件，第二AR信息控件显示有第一故事角色的搜证信息。
在一个实施例中，用户B在用户C角色三的婢女身上发现伤痕，这一搜证信息显示于用户C周侧的AR信息控件上。
该搜证信息可选择公开或私有，若选择公开，则该故事场景中的其他故事角色也可查看；若选择私有，则该故事场景中的其他故事角色无法查看。
针对谜题推理任务：
在一些实施例中，用户可以使用终端基于AR视频流完成谜题推理任务，本实施例提供的基于AR视频流完成谜题推理任务可分为以下两种情况：
第一种情况，基于时间线控件：
步骤a：显示故事场景对应的时间线控件，时间线控件上按照时间顺序显示有第一故事角色的角色信息和搜证信息中的至少一种；
时间线控件是可以按照时间顺序显示角色信息或搜证信息的控件，该时间顺序可以是真实时间顺序，或者，是故事场景的时间顺序。
示例性的，用户A在AR终端显示的时间线控件上将已获取的关于角色二的信息按照故事场景中的时间顺序进行排序。
步骤b：响应于时间线控件上的推理操作，在时间维度上对故事场景对应的推理任务进行推理。
基于时间线控件的推理操作包括但不限于在已获得的角色信息或搜证信息中的时间线上寻找或推理出疑点。
示例性的，用户A在按照故事场景中的时间顺序显示的角色二的信息中，找到角色二在案发时的不在场证明失效的证据。
第二种情况，基于虚拟地图控件：
步骤c：显示故事场景对应的虚拟地图控件，虚拟地图控件上按照地理位置显示有第一故事角色的角色信息和搜证信息中的至少一种；
虚拟地图控件是可以按照地理位置显示角色信息或搜证信息的控件，该虚拟地图可以是真实地理位置，或者，是故事场景的地理位置。
示例性的，用户A在AR终端显示的虚拟地图控件上将已获取的关于角色三的信息按照故事场景中的地理位置进行显示。
步骤d：响应于虚拟地图控件上的推理操作，在空间维度上对故事场景对应的推理任务进行推理。
基于虚拟地图控件的推理操作包括但不限于在已获得的角色信息或搜证信息的地理位置中寻找或推理出疑点。
示例性的，用户A在按照故事场景中的地理位置显示的角色三的信息中，找到角色三去过案发现场的证据。
图11示出了本申请一个示例性实施例提供的基于故事场景的人机互动方法的流程图。该方法可以由终端和服务器来执行。该方法包括：
步骤1101：终端获取摄像头采集的实景视频流；
终端上设置有摄像头。在开启推理任务后，终端上的摄像头对前方环境进行拍摄，得到 实景视频流。实景视频流包括按序排列的多个实景视频帧(简称视频帧)。
可选地，实景视频流可以是经过编码压缩后的视频流。
步骤1102：终端将实景视频流发送至服务器；
终端通过无线网络或有线网络将实景视频流发送至服务器。
步骤1103：服务器接收终端上报的实景视频流；
步骤1104：服务器对实景视频流中的实景视频帧进行图像语义识别，得到实景视频帧中的背景区域和前景人物区域；前景人物区域对应实景人物；
服务器内存储有图像语义分割模型。服务器将实景视频流中的上一视频帧的语义分割结果和当前视频帧输入图像语义分割模型，得到当前视频帧的语义分割结果，语义分割结果包括背景区域和前景人物区域。
可选地，在对实景视频流中的第1个视频帧进行处理时，服务器将参考分割结果和第1个视频帧输入图像语义分割模型，得到第1个视频帧的语义分割结果。该参考分割结果可以是预设的分割结果，或者使用其它模型对第1个视频帧进行语义分割后的粗略分割结果，或者，该参考分割结果是空白的分割结果，本申请对此不加以限定。
可选地，在对实景视频流中的除第1个视频帧之外的其它视频帧进行处理时，服务器将第i-1个视频帧的分割结果和第i个视频帧输入图像语义分割模型，得到第i个视频帧的语义分割结果。
在一个示例中，该图像语义分割模型能够输出2个语义分类：背景区域和前景人物区域；在一个示例中，该图像语义分割模型能够输出3个语义分类：背景区域、前景人物区域中的脸部区域和非脸部区域；在再一个示例中，该图像语义分割模型能够输出多个语义分类：背景区域和前景人物区域中的多个子区域，该多个子区域包括脸部区域、躯干区域、四肢区域、手掌区域、手指区域、骨干关键点中的至少两种。本申请对图像语义分割模型的语义分割能力不加以限定。
步骤1105：服务器对背景区域中的画面内容处理得到增强现实AR背景区域，以及对前景人物区域中的画面内容处理得到AR人物区域；AR背景区域显示故事场景的场景画面，AR人物区域显示穿戴有AR装扮的实景人物，AR装扮对应故事场景中的故事角色；
在一个推理任务中存在至少一个故事场景。一个故事场景中存在至少一个故事角色。每个故事场景对应有场景素材，每个故事角色对应有角色素材。
故事场景的场景素材包括但不限于：自然环境素材、人文建筑素材、室外装饰素材、室内装饰素材、家具素材、环境道具素材中的至少一种。
故事角色的角色素材包括但不限于：首饰素材、脸妆素材、上衣素材、裤装素材、连衣裙素材、鞋子素材、手持道具素材、交通工具素材或坐骑素材等等。比如一个古代剑客的故事角色的角色素材包括玉簪素材、门派服装、宝剑素材等。又比如一个西部牛仔的故事角色的角色素材包括牛仔帽素材、衬衫素材、牛仔裤素材、马匹素材和手枪素材等等。
在识别出实景视频帧中的背景区域和前景人物区域后，服务器基于故事场景的场景素材对背景区域进行替换或融合，得到AR背景区域；以及基于第一故事角色的角色素材对非脸部区域进行替换或融合，得到AR人物区域。
在一个示例中，服务器在不考虑背景区域的任何实物信息的情况下，直接使用场景素材对背景区域进行替换得到AR背景区域，比如将实景中的办公室背景替换为门派大山的背景；在另一个示例中，服务器在考虑背景区域中的三维结构信息的情况下，保留背景区域中的原始主体结构的情况下，使用场景素材中的表面贴图对背景区域中的环境进行重新渲染，得到基于原始主体结构的个性化的AR背景区域，比如将实景中的房间重新渲染为一个贵妃的起居室。
在一个示例中，服务器基于第一故事角色的角色素材对非脸部区域的服装进行替换得到AR人物区域；或者，服务器基于第一故事角色的角色素材对非脸部区域增加虚拟首饰或虚 拟道具，得到AR人物区域。
可选地，由于每个实景人物绑定的故事角色可以各不相同。
在一些实施例中，同一个故事角色在同一个故事场景的不同时间段具有不同的AR装扮；或，同一个故事角色在同一个故事场景的不同地点具有不同的AR装扮；或，同一个故事角色在不同故事场景具有不同的AR装扮；或，同一个故事角色在不同故事场景的不同时间段具有不同的AR装扮；或，同一个故事角色在不同故事场景的不同地点具有不同的AR装扮。
步骤1106：服务器基于将AR背景区域和AR人物区域合并后的AR视频帧，得到AR视频流；
可选地，AR视频帧与实景视频帧存在一一对应关系。对同一个实景视频帧进行处理后的AR背景区域和AR人物区域进行合并，得到与该实景视频帧对应的AR视频帧。
服务器将各个AR视频帧按照时间顺序进行拼接，得到AR视频流。可选地，服务器还将AR视频流进行编码压缩，以减少数据传输时的网络带宽占用。
步骤1107：服务器向终端发送AR视频流；
服务器通过无线网络或有线网络将实景视频流发送至终端。
步骤1108：终端基于AR视频流完成故事场景对应的推理任务。
综上所述，本实施例提供的方法，通过由位于云端的服务器承担图像语义分割的计算任务，能够使得终端本地的计算资源消耗大幅度降低，具有更为流畅的AR体验。
另外，结合参考图12，本实施例在传统静态语义分割的基础上，实现了动态视频语义分割。在传统静态语义分割中，将待分类的图像输入至图像语义分割模型中进行语义分割，得到语义分割结果。而在本实施例中，将上一视频帧的语义分割结果和当前视频帧输入至图像语义分割模型中进行语义分割，得到当前视频帧的语义分割结果。由于在当前视频帧的图像语义分割任务中，引入了上一视频帧的语义分割结果作为参考信息，利用上一视频帧和当前视频帧在时域上的高度关联性，能够加快图像语义分割任务的计算速度，缩短对当前视频帧进行图像语义分割时的耗时，提高对当前视频帧进行图像语义分割时的准确度。
在一些实施例中，该图像语义分割模型是基于基本样本库进行训练得到的。基本样本库包括:上一样本视频帧的语义分割标签、当前样本视频帧和当前样本视频帧的语义分割标签。其中，上一样本视频帧是在样本视频帧中位于当前样本视频帧之前的视频帧，若当前样本视频帧是首帧，则上一视频帧可使用仿射变换或薄板插样后的当前样本视频帧代替。上一样本视频帧的语义分割标识和当前样本视频帧的语义分割标签可以是人为标注的抠图样本掩码，或者，由传统的静态图像语义分割模型进行语义分割后的抠图样本掩码。
仿射变换(affine transformation)：是一种图像变换方式，描述一种二维坐标点的线性变换。仿射变换可将二维坐标点进行一次线性变换并接上一次平移。在本申请实施例中，仿射变换可以模拟实景人物的移动。
薄板插样(Thin Plate Spline，TPS，也可称为薄板样条插值)：是一种二维插值方法，用于对图像上的控制点进行偏移，以达到通过控制点对图像进行特定形变的目的。在本申请实施例中，薄板插样可以模拟摄像头的快速抖动。
在一些实施例中，该图像语义分割模型是基于采用基本样本库和增强样本库训练得到的；
基本样本库包括:上一样本视频帧的语义分割标签、当前样本视频帧和当前样本视频帧的语义分割标签；
增强样本库包括：上一样本视频帧的语义分割标签、当前增强视频帧和当前增强视频帧的语义分割标签；
其中，当前增强视频帧是对当前样本视频帧进行仿射变换或薄板插样得到的，当前增强视频帧的语义分割标签是对当前样本视频帧的语义分割标签进行仿射变换和/或薄板插样得到的。
在基本样本库中的样本有限的情况下，为了模仿实景人物移动或摄像头快速抖动等场景， 将同一对“当前样本视频帧+语义分割标签”进行相同方式的仿射变换或薄板插样后，得到新的一对“当前增强视频帧+语义分割标签”，从而形成了增强样本。在进行多次不同的仿射变换或薄板插样后，能够得到多个增强样本从而形成增强样本库。
可选地，服务器对同一对当前样本视频帧和当前样本视频帧语义分割标签中的背景区域进行相同方式的仿射变换或薄板插样后，得到第一增强样本；服务器对同一对当前样本视频帧和当前样本视频帧语义分割标签中的前景人物区域进行相同方式的仿射变换或薄板插样后，得到第二增强样本。
在基于图11的可选实施例中，上述图像语义分割模型可以采用全卷积网(Fully Convolutional Networks，FCN)网络来实现。
图像语义分割模型需要判断图像每个像素点的类别。即图像语义分割是像素级别的。以往用于语义分割的卷积网络(Convolutional Neural Network，CNN)，每个像素点用包围自身的区域类别进行标注，但是这种方法不管是在速度上还是精度上都有很大的缺陷。FCN基于CNN改进而来。
如图13所示，FCN整体的网络结构分为两个部分:全卷积部分和反卷积部分。其中全卷积部分借用了一些经典的CNN网络(如AlexNet，VGG，GoogLeNet等，AlexNet是2012年推出的一种神经网络，VGG指Visual Geometry Group network，视觉几何小组网络，GoogLeNet是2014年提出的一种全新的深度学习结构)，并把最后的全连接层换成卷积层，该卷积层用于提取特征形成热点图；反卷积部分则是将小尺寸的热点图上采样得到原始尺寸的语义分割图像。FCN网络的输入可以为任意尺寸的彩色图像，输出与输入尺寸相同，通道数为n(目标类别数)+1(背景)。FCN网络在CNN的卷积部分不用全连接层而是替换成卷积层的目的是允许输入的图像为超过某一尺寸的任意大小。
由于在卷积过程中，热点图变得很小(比如长宽变为原图像的7/50)，为了得到原图像大小的稠密像素预测，需要进行上采样。一个直观的想法是进行双线性插值，而双线性插值很容易用反向卷积通过固定的卷积核来实现。反向卷积又可以被称为反卷积，也通常被称为转置卷积。
如果利用之前提到的上采样对最后一层的特征图进行上采样的到原图大小的分割，由于最后一层的特征图太小，导致会损失很多细节。因而提出增加跨越(Skips)结构将最后一层的预测(有更富的全局信息)和更浅层(有更多的局部细节)的预测结合起来，这样可以在遵守全局预测的同时进行局部预测。
将底层(步长32)的预测(FCN-32s)进行2倍的上采样得到原尺寸的图像，并与从池4层(步长16)进行的预测融合起来(相加)，这一部分的网络被称为FCN-16s。随后将这一部分的预测再进行一次2倍的上采样并与从池3层得到的预测融合起来，这一部分的网络被称为FCN-8s。结合参考图14，图14示出了FCN-32s、FCN-16s、FCN-8s和真实样本之间的效果对比。可以看出，FCN-8s的语义分割结果最接近真实样本，FCN-8s的语义分割结果强于FCN-16s的语义分割结果、FCN-16s的语义分割结果强于FCN-32s的语义分割结果。
FCN对图像进行像素级的分类，从而解决了语义级别的图像分割问题。与经典的CNN在卷积层之后使用全连接层得到固定长度的特征向量进行分类不同，FCN可以接受任意尺寸的输入图像，采用反卷积层对最后一个卷积层的特征图进行上采样,使它恢复到输入图像相同的尺寸，从而可以对每个像素都产生了一个预测,同时保留了原始输入图像中的空间信息,最后在上采样的特征图上进行逐像素分类。最后逐个像素计算分类损失,相当于每一个像素对应一个训练样本。
简单的来说，FCN与CNN的区域在把于CNN最后的全连接层换成卷积层，输出的是一张已经标注好的图片。
图15示出了搜证阶段用户E的AR终端显示的AR画面300，该画面上包括AR背景区域301、AR人物区域302和AR信息控件。AR背景区域301显示搜证阶段的虚拟场景和虚 拟道具305，AR人物区域302显示穿戴有AR装扮303的实景人物，AR装扮303对应故事场景中的故事角色，AR信息控件304显示故事角色的角色信息或搜证信息，可位于故事角色绑定的实景人物周侧或搜证信息获取的位置。
示例性的，AR背景区域301显示为城外山脚的柳树下。AR人物区域302显示穿戴古风AR服饰装束的用户B，并且用户B正在弹奏AR虚拟古琴。用户E已获取的第二故事角色的角色信息显示于用户B周侧的AR信息控件304上。用户E获取的第二故事角色的私有信息，用户A、用户B、用户C、用户D无法查看，用户E获取的第二故事角色的公开信息，用户A、用户B、用户C、用户D均可以查看。用户E对虚拟古琴搜证操作后获取的第二故事角色的搜证信息，显示在位于该搜证信息获取位置的AR信息控件上，该信息可选择公开或私有，若选择公开，则该故事场景中的其他故事角色也可查看；若选择私有，则该故事场景中的其他故事角色无法查看。
图16示出了本申请一个示例性实施例提供的基于故事场景的人机互动方法的游戏场景流程图。本实施例以该方法由图3所示的终端执行来举例说明。该终端具有摄像头。该方法包括：选择推理任务1601、选择故事角色1602、读剧本1603、介绍阶段1604、公聊阶段1605、私聊阶段1606、搜证阶段1607和结案阶段1608。
选择推理任务1601：用户在终端显示的至少两个候选故事场景的推理任务中选择任一推理任务；
示例性的，用户A在枪战谍战场景、仙侠修仙场景、西部牛仔场景和古墓探险场景中选择现枪战谍战场景。
选择故事角色1602：用户选择故事场景后，在终端显示的至少两个候选故事角色中选择任一故事角色，终端摄像头采集用户图像并人脸识别，对用户进行AR换装，将故事角色与用户匹配；
示例性的，用户A选择第一故事角色特工A并完成角色绑定，完成AR换装。
读剧本1603：用户阅读已选择的故事场景的背景信息，了解该故事场景的背景、时间、任务及已绑定的故事角色基本信息等；
介绍阶段1604：用户向同个故事场景中的其他故事角色介绍自己，并获取同个故事场景中的其他故事角色的基本信息，该信息为公开信息，信息获取方式包括：从服务器获取、语音录入、OCR扫描和键盘输入中的至少一种，该信息显示于AR信息控件中，用户可以通过终端设备扫描对应故事角色查看该AR信息；
示例性的，用户A向同个故事场景中的用户自我介绍，并从服务器中获取用户B和用户C的基本信息，该信息显示在位于用户B和用户C周侧的AR信息控件中，该信息为公开信息，用户A、用户B和用户C可通过终端设备扫描对应故事角色查看其基本信息。
公聊阶段1605：同个故事场景中的所有用户进行信息交换，用户可获取同个故事场景中的故事角色的扩展信息，该信息为公开信息，信息获取方式包括：从服务器获取、语音录入、OCR扫描和键盘输入中的至少一种，该信息显示于AR信息控件中，用户可以通过终端设备扫描对应故事角色查看该AR信息；
示例性的，用户A在公聊阶段以OCR扫描的方式获取用户B和用户C的近三天日程表，该信息显示在位于用户B和用户C周侧的AR信息控件中，该信息为公开信息，用户A、用户B和用户C可以通过终端设备扫描用户B和用户C绑定的故事角色查看该扩展信息。
私聊阶段1606：同个故事场景中仅某两个故事角色进行信息交换，用户获取与之私聊的故事角色的扩展信息，该信息为私有信息，信息获取方式包括：从服务器获取、语音录入、OCR扫描和键盘输入中的至少一种，该信息显示于AR信息控件中，用户可以通过终端设备扫描对应故事角色查看该AR信息；
示例性的，用户A与用户B私聊，以文字输入的方式获取用户B绑定的第二故事角色的扩展信息：用户B的工具来源为用户C，该信息显示在位于用户B周侧的AR信息控件中， 该信息仅用户A具有查看权限，用户A可以通过终端设备扫描用户B绑定的第二故事角色查看该扩展信息。
搜证阶段1607：故事角色对同个故事场景中的其他故事角色相关的虚拟场景或虚拟道具进行搜证操作，获取其他故事角色的搜证信息，该信息可以选择公开或私有，若选择公开，则该故事场景中的其他故事角色也可查看；若选择私有，则该故事场景中的其他故事角色无法查看。用户可以通过终端设备扫描对应故事角色或虚拟道具查看该搜证信息。
示例性的，用户A对用户C绑定的第三故事角色的书桌进行搜证操作，获取工具买卖清单，该信息显示在位于该书桌位置的AR信息控件上，用户A选择公开该搜证信息，用户A、用户B和用户C均通过终端设备扫描用户C绑定的第三故事角色或该书桌查看该搜证信息。
结案阶段1608：用户A、用户B和用户C进行投票，投票结果为用户B为目标人物，推理结果正确，推理任务完成，结案。
综上所述，本实施例提供的方法利用终端执行人机互动和推理任务，利用人脸识别和AR换装将用户与故事角色进行绑定，可以从服务器获取、语音录入、OCR扫描和键盘输入中的至少一种来获取角色信息、搜证信息，游戏操作简单方便，提供更为沉浸的游戏体验。
图17示出了本申请另一个示例性实施例提供的计算机系统的结构框图。本实施例以该方法由图3所示的计算机系统执行来举例说明。该系统包括：客户端1701、后台服务1702、架构引擎1703、数据存储1704、运行环境1705。
客户端1701：指终端上支持AR互动和推理任务的Android或iOS应用程序。该终端可以是智能手机、平板电脑、电子书阅读器、膝上便携计算机、台式计算机和AR眼镜中等具有摄像头的电子设备。客户端1701支持终端进行剧本选择操作、故事角色选择操作和人脸信息录入。客户端1701支持显示AR场景、AR着装和AR信息中的至少一种AR功能。客户端1701支持信息记录功能，可通过OCR输入、语音输入和键盘输入中的至少一种方式记录信息；
后台服务1702：由服务器320提供的支持客户端1701执行的数据服务、AR服务和智能输入服务中的至少一种后台服务，对客户端1701的请求实现拦截和响应，对客户端1701的所有请求进行筛选、过滤或调用第三方接口，对信息进行包装后再传回客户端1701；
架构引擎1703：通过GIN框架(一种网页框架)执行对应用程序的启动、请求参数的处理和响应格式的渲染等操作，通过AR引擎处理AR功能的操作，通过AI引擎处理有关机器学习的计算操作；
数据存储1704：包括存储一般信息的MySQL数据库(一种关系数据库管理系统)和用户存储海量用户日志和用户图库的MongoDB数据库(是一个基于分布式文件存储的数据库)，两种数据库各自独立存储，都通过Hadoop(是一个分布式系统基础架构)实现集群分布式部署存储，利用分布式关系型数据库(Distribute Relational Database Service，DRDS)作为中间件实现弹性存储；
运行环境1705：后台服务1702利用云计算平台承担基于客户端数据集的判别器和生成器的训练任务，通过人脸识别和图像语义识别将实景视频流替换为AR视频流，再将图像传回支持AR互动和推理任务的Android或iOS客户端1701，为用户提供更流畅、更沉浸的AR体验。
本领域技术人员可以理解，图17中示出的计算机结构并不构成对计算机系统的限定，可以包括比图示更多或更少的结构，或者组合某些结构，或者采用不同的结构布置。
图18示出了本申请一个示例性实施例提供的基于故事场景的人机互动装置的框图。该装置包括：获取模块1802、显示模块1804、处理模块1806、和互动模块1808。
获取模块1802：用于执行上述实施例中图2所示的步骤220。
显示模块1804：用于执行上述实施例中图2所示的步骤240。
显示模块1804用于在AR视频流中显示第一AR信息，第一AR信息用于显示第一故事 角色的角色信息。显示模块1804用于在AR视频流中显示第二AR信息，第二AR信息用于显示第一故事角色的搜证信息。
在一个可选的设计中，显示模块1804用于在AR视频流中显示位于实景人物周侧的第一AR信息控件，第一AR信息控件显示有第一AR信息。显示模块1804用于在AR视频流中显示第二AR信息控件，第二AR信息控件用于显示第二AR信息。
在一个可选的设计中，显示模块1804用于在AR人物区域的第二脸部区域中，显示实景人物未佩戴有AR装置的第二脸部画面。
处理模块1806：用于执行上述实施例中图2所示的步骤241-步骤247中的至少一个步骤。
互动模块1808：用于执行上述实施例中图2所示的步骤260、图8所示的步骤266a-步骤268a、图9所示的步骤266c-步骤268b、图10所示的步骤266e-步骤268c中的至少一个步骤。
在一个可选的设计中，互动模块1808用于如下方式中的至少一种：从服务器获取第一故事角色的角色信息；通过语音录入方式获取第一故事角色的角色信息；通过光学字符识别OCR扫描方式获取第一故事角色的角色信息；通过键盘输入方式获取第一故事角色的角色信息。
在一个可选的设计中，装置还包括：上传模块，用于接收AR装扮的上传操作；响应于上传操作，将本地创建的AR装扮上传至服务器中。
在一个可选的设计中，装置还包括：自定义模块，用于接收AR装扮的自定义操作；响应于自定义操作，将自定义的AR装扮上传至服务器中。
需要说明的是，本实施例仅对模块的功能做简要说明，具体内容可以参考上述实施例中的内容。
图19示出了本申请一个示例性实施例提供的基于故事场景的人机互动装置的框图。该装置包括：接收模块1902、处理模块1904、和互动模块1906。
接收模块1902，用于执行上述实施例中图11所示的步骤1103。
处理模块1904，用于执行上述实施例中图11所示的步骤1104-步骤1106中的至少一个步骤。
互动模块1906，用于基于AR视频流完成故事场景对应的推理任务。
互动模块1906基于处理模块1904处理得到的AR视频流，完成故事场景对应的信息获取任务、证据搜集任务和谜题推理任务中的至少一种。
在一个可选的设计中，互动模块1906用于获取第一故事角色的角色信息；在一个可选的设计中，互动模块1906用于获取第一故事角色的搜证信息。
在一个可选的设计中，互动模块1906用于响应于时间线控件上的推理操作，在时间维度上对故事场景对应的推理任务进行搜证；或，互动模块1906用于响应于虚拟地图控件上的推理操作，在空间维度上对故事场景对应的推理任务进行搜证；或，互动模块1906用于响应于对虚拟场景中指定位置的查看操作，获取第一故事角色在虚拟场景中的第一搜证信息；或，互动模块1906用于响应于对虚拟道具的互动操作，获取第一故事角色在虚拟道具上关联的第二搜证信息；或，互动模块1906用于响应于对NPC故事角色的互动操作，获取第一故事角色的第三搜证信息。
需要说明的是，本实施例仅对模块的功能做简要说明，具体内容可以参考上述实施例中的内容。
需要说明的是：上述实施例提供的装置在基于故事场景的人机互动时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。且具体实现过程详见方法实施例，这里不再赘述。
图20示出了本申请一个示例性实施例提供的终端2000的结构框图。该终端2000可以是智能手机、平板电脑、电子书阅读器、膝上便携计算机、台式计算机和AR眼镜中等具有摄像 头的电子设备。终端2000还可能被称为用户设备、便携式终端、膝上型终端、台式终端等其他名称。
通常，终端2000包括有：处理器2001和存储器2002。
处理器2001可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器2001可以采用DSP(Digital Signal Processing，数字信号处理)、FPGA(Field－Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器2001也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器2001可以在集成GPU(Graphics Processing Unit，图像处理器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器2001还可以包括AR处理器，该AR处理器用于处理有关增强现实的计算操作。一些实施例中，处理器2001还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。
存储器2002可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器2002还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器2002中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器2001所执行以实现本申请中方法实施例提供的基于故事场景的人机互动方法。
在一些实施例中，终端2000还可选包括有：外围设备接口2003和至少一个外围设备。处理器2001、存储器2002和外围设备接口2003之间可以通过总线或信号线相连。各个外围设备可以通过总线、信号线或电路板与外围设备接口2003相连。具体地，外围设备可以包括：射频电路2004、显示屏2005、摄像头组件2006、音频电路2007、电源2008中的至少一种。
外围设备接口2003可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器2001和存储器2002。
射频电路2004用于接收和发射RF(Radio Frequency，射频)信号，也称电磁信号。
显示屏2005用于显示UI(User Interface，用户界面)。
摄像头组件2006用于采集图像或视频。
音频电路2007可以包括麦克风和扬声器。
电源2008用于为终端2000中的各个组件进行供电。
在一些实施例中，终端2000还包括有一个或多个传感器2009。该一个或多个传感器2009包括但不限于：加速度传感器2010、陀螺仪传感器2011、压力传感器2012、光学传感器2013、以及接近传感器2014。
加速度传感器2010可以检测以终端2000建立的坐标系的三个坐标轴上的加速度大小。
陀螺仪传感器2011可以检测终端2000的机体方向及转动角度，陀螺仪传感器2011可以与加速度传感器2011协同采集用户对终端2000的3D动作。
压力传感器2012可以设置在终端2000的侧边框和/或显示屏2005的下层。
光学传感器2013用于采集环境光强度。
接近传感器2014，也称距离传感器，通常设置在终端2000的前面板。接近传感器2014用于采集用户与终端2000的正面之间的距离。所述存储器还包括一个或者一个以上的程序，所述一个或者一个以上程序存储于存储器中，所述一个或者一个以上程序包含用于进行本申请实施例提供的基于虚拟环境的信号显示方法。
本领域技术人员可以理解，图20中示出的结构并不构成对终端2000的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
在示例性实施例中，还提供了一种终端，所述终端包括处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集。所述至少一条指令、至少一段程序、 代码集或指令集经配置以由所述处理器执行，以实现上述基于故事场景的人机互动方法。
在示例性实施例中，还提供了一种服务器2100，所述服务器2100包括处理器2101和存储器2102。图21示出了本申请一个示例性实施例提供的服务器2100的结构框图。
通常，服务器2100包括有：处理器2101和存储器2102。
处理器2101可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器2101可以采用数字信号处理(Digital Signal Processing，DSP)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)、可编程逻辑阵列(Programmable Logic Array，PLA)中的至少一种硬件形式来实现。处理器2101也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称中央处理器(Central Processing Unit，CPU)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器2101可以在集成有图像处理器(Graphics Processing Unit，GPU)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器2101还可以包括人工智能(Artificial Intelligence，AI)处理器，该AI处理器用于处理有关机器学习的计算操作。
存储器2102可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器2102还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器2102中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器2101所执行以实现本申请中方法实施例提供的三维空间的全局光照计算方法。
在一些实施例中，服务器2100还可选包括有：输入接口2103和输出接口2104。处理器2101、存储器2102和输入接口2103、输出接口2104之间可以通过总线或信号线相连。各个外围设备可以通过总线、信号线或电路板与输入接口2103、输出接口2104相连。输入接口2103、输出接口2104可被用于将输入/输出(Input/Output，I/O)相关的至少一个外围设备连接到处理器2101和存储器2102。在一些实施例中，处理器2101、存储器2102和输入接口2103、输出接口2104被集成在同一芯片或电路板上；在一些其他实施例中，处理器2101、存储器2102和输入接口2103、输出接口2104中的任意一个或两个可以在单独的芯片或电路板上实现，本申请实施例对此不加以限定。
本领域技术人员可以理解，上述示出的结构并不构成对服务器2100的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
在示例性实施例中，还提供了一种计算机可读存储介质，所述存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或所述指令集在被终端的处理器执行时实现上述虚拟环境中的视角切换方法。可选地，上述计算机可读存储介质可以是ROM(Read-Only Memory，只读存储器)、RAM(Random Access Memory，随机存取存储器)、CD-ROM(Compact Disc Read-Only Memory，只读光盘)、磁带、软盘和光数据存储设备等。
在示例性实施例中，还提供了一种计算机程序产品，所述计算机程序产品存储有计算机程序，所述计算机程序由处理器加载并执行以实现如上所述的基于故事场景的人机互动方法。

Claims (33)

  1. 一种基于故事场景的人机互动方法，其中，所述方法由具有摄像头的终端执行，所述方法包括：
    获取所述摄像头采集的实景视频流，所述实景视频流的视频画面包括背景区域和前景人物区域，所述前景人物区域对应实景人物；
    基于所述实景视频流显示增强现实AR视频流，所述AR视频流的视频画面包括AR背景区域和AR人物区域，所述AR背景区域显示所述故事场景的场景画面，所述AR人物区域显示穿戴有AR装扮的所述实景人物，所述AR装扮对应所述故事场景中的故事角色；
    响应于交互操作，改变所述AR视频流的显示内容；
    基于改变后的显示内容，完成所述故事场景对应的推理任务；
    其中，所述AR背景区域是对所述背景区域中的画面内容处理得到的，所述AR人物区域是对所述前景人物区域中的画面内容处理得到的。
  2. 根据权利要求1所述的方法，其中，所述基于所述实景视频流显示AR视频流，包括：
    在所述实景视频流中的所述实景人物与所述故事场景中的第一故事角色绑定的情况下，显示所述AR视频流；所述AR视频流中的所述AR人物区域显示穿戴有第一AR装扮的所述实景人物；
    其中，所述第一AR装扮包括所述第一故事角色在所述故事场景中对应的服装。
  3. 根据权利要求2所述的方法，其中，所述在所述实景视频流中的所述实景人物与所述故事场景中的第一故事角色绑定的情况下，显示所述AR视频流，包括：
    对所述实景视频流中的视频帧进行图像语义识别，得到所述背景区域和所述前景人物区域，所述前景人物区域包括脸部区域和非脸部区域；
    对所述脸部区域进行脸部识别，得到所述实景人物的人物身份；
    基于所述实景人物的人物身份，确定所述实景人物在所述故事场景中绑定的第一故事角色；
    基于所述故事场景的场景素材对所述背景区域进行替换或融合，得到所述AR背景区域；以及基于所述第一故事角色的角色素材对所述非脸部区域进行替换或融合，得到所述AR人物区域；
    基于所述AR背景区域和所述AR人物区域，显示所述AR视频流。
  4. 根据权利要求1至3任一所述的方法，其中，所述前景人物区域的第一脸部区域中显示有所述实景人物佩戴有AR终端的第一脸部画面；所述方法还包括：
    在所述AR人物区域的第二脸部区域中，显示所述实景人物未佩戴有所述AR终端的第二脸部画面。
  5. 根据权利要求4所述的方法，其中，所述在所述AR人物区域的第二脸部区域中，显示所述实景人物未佩戴有所述AR终端的第二脸部画面，包括：
    将所述第一脸部画面和所述实景人物的样本人脸数据输入至生成式网络进行图像重建，得到所述实景人物未佩戴有所述AR终端的第二脸部画面；
    在所述AR人物区域的第二脸部区域中，显示所述第二脸部画面。
  6. 根据权利要求2或3所述的方法，其中，所述基于改变后的显示内容，完成所述故事场景对应的推理任务，包括：
    基于所述改变后的显示内容，完成所述故事场景对应的互动推理任务，所述互动推理任务是与所述故事场景的场景画面互动的任务，和/或，所述互动推理任务是与所述故事场景中的故事角色互动的任务。
  7. 根据权利要求2或3所述的方法，其中，所述基于所述AR视频流完成所述故事场景对应的推理任务，包括：
    基于所述AR视频流中显示的AR信息，完成所述故事场景对应的信息获取任务、证据搜集任务和谜题推理任务中的至少一种。
  8. 根据权利要求7所述的方法，其中，所述基于所述AR视频流中显示的AR信息，完成所述故事场景对应的信息获取任务，包括：
    获取所述第一故事角色的角色信息；
    在所述AR视频流中显示第一AR信息，所述第一AR信息用于将所述角色信息关联至所述第一故事角色对应的实景人物进行显示。
  9. 根据权利要求8所述的方法，其中，所述获取所述第一故事角色的角色信息，包括如下方式中的至少一种：
    从服务器获取所述第一故事角色的角色信息；
    通过语音录入方式获取所述第一故事角色的角色信息；
    通过光学字符识别OCR扫描方式获取所述第一故事角色的角色信息；
    通过键盘输入方式获取所述第一故事角色的角色信息。
  10. 根据权利要求8所述的方法，其中，所述第一故事角色的角色信息包括公开角色信息和私有角色信息中的至少一种；
    所述公开角色信息是参与所述故事场景中的至少两个故事角色具有查看权限的角色信息；
    所述私有角色信息是所述终端对应的第二故事角色具有查看权限的角色信息。
  11. 根据权利要求8所述的方法，其中，所述在所述AR视频流中显示第一AR信息，包括：
    在所述AR视频流中显示位于所述实景人物周侧的第一AR信息控件，所述AR信息控件显示有所述第一AR信息。
  12. 根据权利要求7所述的方法，其中，所述基于所述AR视频流中显示的AR信息，完成所述故事场景对应的证据搜集任务，包括：
    获取所述第一故事角色的搜证信息；
    在所述AR视频流中显示第二AR信息，所述第二AR信息用于显示所述第一故事角色的搜证信息。
  13. 根据权利要求12所述的方法，其中，所述获取所述第一故事角色的搜证信息，包括：
    显示与所述第一故事角色相关的虚拟场景；响应于对所述虚拟场景中指定位置的查看操作，获取所述第一故事角色在所述虚拟场景中的第一搜证信息；
    或，
    显示与所述第一故事角色相关的虚拟道具；响应于对所述虚拟道具的互动操作，获取所述第一故事角色在所述虚拟道具上关联的第二搜证信息；
    或，
    显示与所述第一故事角色相关的非玩家角色NPC故事角色；响应于对所述NPC故事角色的互动操作，获取所述第一故事角色的第三搜证信息。
  14. 根据权利要求12所述的方法，其中，所述在所述AR视频流中显示第二AR信息，包括：
    在所述AR视频流中显示位于所述第一故事角色对应的实景人物周侧的第二AR信息控件，所述第二AR信息控件显示有所述第二AR信息；
    或，
    在所述AR视频流中，在所述搜证信息的获取位置显示第二AR信息控件，所述第二AR信息控件显示有所述第二AR信息。
  15. 根据权利要求7所述的方法，其中，所述基于所述AR视频流中显示的AR信息，完成所述故事场景对应的谜题推理任务，包括：
    显示所述故事场景对应的时间线控件，所述时间线控件上按照时间顺序显示有所述第一 故事角色的角色信息和搜证信息中的至少一种；响应于所述时间线控件上的推理操作，在时间维度上对所述故事场景对应的推理任务进行推理；
    或，显示所述故事场景对应的虚拟地图控件，所述虚拟地图控件上按照地理位置显示有所述第一故事角色的角色信息和搜证信息中的至少一种；响应于所述虚拟地图控件上的推理操作，在空间维度上对所述故事场景对应的推理任务进行推理。
  16. 根据权利要求1至15任一所述的方法，其中，所述响应于交互操作，改变所述AR视频流的显示内容，包括如下至少之一：
    响应于与所述AR背景区域中虚拟物品的物品交互操作，改变所述AR背景区域的显示内容；
    响应于与所述AR人物区域中所述实景人物的人物交互操作，改变所述实景人物的角色信息；
    响应于所述故事场景的场景切换操作，改变所述故事场景的场景画面；
    响应于所述故事场景的故事情节触发操作，改变所述AR视频流中与所述故事情节有关的显示内容。
  17. 根据权利要求16所述的方法，其特征在于，所述响应于与所述AR背景区域中虚拟物品的物品交互操作，改变所述AR背景区域的显示内容，包括如下至少之一：
    响应于与所述AR背景区域中所述虚拟物品的所述物品交互操作，在所述AR背景区域中显示故事线索；
    响应于与所述AR背景区域中所述虚拟物品的所述物品交互操作，更新显示所述AR背景区域中的所述虚拟物品；
    响应于与所述AR背景区域中所述虚拟物品的所述物品交互操作，更新所述AR背景区域中的所述故事场景的场景画面。
  18. 根据权利要求16所述的方法，其特征在于，所述物品交互操作包括物品触摸操作、物品抓举操作、物品使用操作、物品检查操作、手势指认操作、眼球锁定操作、眼球滑动操作中的至少一种。
  19. 根据权利要求16所述的方法，其特征在于，所述响应于与所述AR人物区域中所述故事角色的人物交互操作，改变所述故事角色的角色信息，包括如下至少之一：
    响应于与所述AR人物区域中所述故事角色的所述人物交互操作，将所述故事角色的第一角色信息更改为第二角色信息；
    响应于与所述AR人物区域中所述故事角色的所述人物交互操作，新增所述故事角色的第三角色信息；
    响应于与所述AR人物区域中所述故事角色的所述人物交互操作，删除所述故事角色的第四角色信息；
    响应于与所述AR人物区域中所述故事角色的所述人物交互操作，增加所述故事角色与其它故事角色之间的角色关系信息。
  20. 根据权利要求16所述的方法，其特征在于，所述人物交互操作包括人物触摸操作、人物抓握操作、人物交谈操作、手势指认操作、眼球锁定操作、眼球滑动操作中的至少一种。
  21. 根据权利要求16所述的方法，其特征在于，所述响应于所述故事场景的场景切换操作，改变所述故事场景的场景画面，包括：
    响应于所述故事场景的所述场景切换操作，将所述故事场景的第一场景画面切换为第二场景画面，所述第一场景画面和所述第二场景画面是不同的场景画面。
  22. 根据权利要求16所述的方法，其特征在于，所述响应于所述故事场景的故事情节触发操作，改变所述AR视频流的显示内容，包括如下至少之一：
    响应于所述故事场景的所述故事情节触发操作，改变所述故事场景的场景画面；
    响应于所述故事场景的所述故事情节触发操作，在所述AR背景区域中新增情节道具；
    响应于所述故事场景的所述故事情节触发操作，更新所述故事角色的角色信息。
  23. 根据权利要求1至15任一所述的方法，其中，所述方法还包括：
    显示至少两个候选故事场景的任务选择控件；
    响应于所述任务选择控件上的选择操作，确定所述至少两个候选故事场景中被选择的所述故事场景。
  24. 根据权利要求1至15任一所述的方法，其中，所述方法还包括：
    显示所述故事场景中的至少两个候选故事角色的角色选择控件；
    响应于所述角色选择控件上的选择操作，确定所述至少两个候选故事角色中被选择的所述故事角色；
    将被选择的所述故事角色与所述终端对应的实景人物的脸部数据进行绑定。
  25. 根据权利要求1至15任一所述的方法，其中，所述方法还包括：
    接收所述AR装扮的上传操作；
    响应于所述上传操作，将本地创建的所述AR装扮上传至服务器中。
  26. 根据权利要求1至15任一所述的方法，其中，所述方法还包括：
    接收所述AR装扮的自定义操作；
    响应于所述自定义操作，将自定义的所述AR装扮上传至服务器中。
  27. 一种基于故事场景的人机互动方法，其中，所述方法由服务器执行，所述方法包括：
    接收终端上报的实景视频流；
    对所述实景视频流中的实景视频帧进行图像语义识别，得到所述实景视频帧中的背景区域和前景人物区域；所述前景人物区域对应实景人物；
    对所述背景区域中的画面内容处理得到增强现实AR背景区域，以及对所述前景人物区域中的画面内容处理得到AR人物区域；所述AR背景区域显示所述故事场景的场景画面，所述AR人物区域显示穿戴有AR装扮的所述实景人物，所述AR装扮对应所述故事场景中的故事角色；
    基于将所述AR背景区域和所述AR人物区域合并后的AR视频帧，得到AR视频流；
    向所述终端发送所述AR视频流，以便所述终端基于所述AR视频流完成所述故事场景对应的推理任务。
  28. 一种基于故事场景的人机互动装置，其中，所述装置具有摄像头，所述装置包括：
    获取模块，用于获取所述摄像头采集的实景视频流，所述实景视频流的视频画面包括背景区域和前景人物区域，所述前景人物区域对应实景人物；
    显示模块，用于基于所述实景视频流显示增强现实AR视频流，所述AR视频流的视频画面包括AR背景区域和AR人物区域，所述AR背景区域显示所述故事场景的场景画面，所述AR人物区域显示穿戴有AR装扮的所述实景人物，所述AR装扮对应所述故事场景中的故事角色；
    互动模块，用于基响应于交互操作，改变所述AR视频流的显示内容；基于改变后的显示内容，完成所述故事场景对应的推理任务；
    其中，所述AR背景区域是对所述背景区域中的画面内容处理得到的，所述AR人物区域是对所述前景人物区域中的画面内容处理得到的。
  29. 一种基于故事场景的人机互动装置，其中，所述装置连接有摄像头，所述装置包括：
    接收模块，用于接收所述摄像头采集的实景视频流，所述实景视频流的视频画面包括背景区域和前景人物区域，所述前景人物区域对应实景人物；
    处理模块，用于将所述实景视频流处理为增强现实AR视频流，所述AR视频流的视频画面包括AR背景区域和AR人物区域，所述AR背景区域显示所述故事场景的场景画面，所述AR人物区域显示穿戴有AR装扮的所述实景人物，所述AR装扮对应所述故事场景中的故事角色；
    互动模块，用于基于所述AR视频流完成所述故事场景对应的推理任务；
    其中，所述AR背景区域是对所述背景区域中的画面内容处理得到的，所述AR人物区域是对所述前景人物区域中的画面内容处理得到的。
  30. 一种终端，其中，所述终端包括：摄像头、处理器和存储器，所述存储器存储有计算机程序，所述计算机程序由所述处理器加载并执行以实现如权利要求1至26任一所述的基于故事场景的人物互动方法。
  31. 一种服务器，其中，所述服务器包括：处理器和存储器，所述存储器存储有计算机程序，所述计算机程序由所述处理器加载并执行以实现如权利要求27所述的基于故事场景的人物互动方法。
  32. 一种计算机可读存储介质，其中，所述计算机可读存储介质存储有计算机程序，所述计算机程序由处理器加载并执行以实现如权利要求1至26任一所述的基于故事场景的人物互动方法，或如权利要求27所述的基于故事场景的人物互动方法。
  33. 一种计算机程序产品，其中，所述计算机可读存储介质存储有计算机程序，所述计算机程序由处理器加载并执行以实现如权利要求1至26任一所述的基于故事场景的人物互动方法，或如权利要求27所述的基于故事场景的人物互动方法。
PCT/CN2022/112406 2022-04-18 2022-08-15 基于故事场景的人机互动方法、装置、设备及介质 WO2023201937A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/204,214 US20230330541A1 (en) 2022-04-18 2023-05-31 Method and apparatus for man-machine interaction based on story scene, device and medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210406828.1 2022-04-18
CN202210406828.1A CN116943191A (zh) 2022-04-18 2022-04-18 基于故事场景的人机互动方法、装置、设备及介质

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/204,214 Continuation US20230330541A1 (en) 2022-04-18 2023-05-31 Method and apparatus for man-machine interaction based on story scene, device and medium

Publications (1)

Publication Number Publication Date
WO2023201937A1 true WO2023201937A1 (zh) 2023-10-26

Family

ID=88419023

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/112406 WO2023201937A1 (zh) 2022-04-18 2022-08-15 基于故事场景的人机互动方法、装置、设备及介质

Country Status (2)

Country Link
CN (1) CN116943191A (zh)
WO (1) WO2023201937A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118247700A (zh) * 2024-03-19 2024-06-25 西安隆腾科技文化有限公司 一种多媒体信息交互方法及系统
CN118433435A (zh) * 2024-06-27 2024-08-02 广州市锐星信息科技有限公司 一种示教直播系统

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118485952B (zh) * 2024-07-16 2024-10-01 江西华广电影视科技有限公司 多场景适应的视频图像智能识别方法及系统

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120122570A1 (en) * 2010-11-16 2012-05-17 David Michael Baronoff Augmented reality gaming experience
CN107596683A (zh) * 2017-09-25 2018-01-19 晋江市博感电子科技有限公司 基于增强现实的童车虚拟游乐方法、装置及系统
CN108421252A (zh) * 2017-02-14 2018-08-21 深圳梦境视觉智能科技有限公司 一种基于ar设备的游戏实现方法和ar设备
US20180349703A1 (en) * 2018-07-27 2018-12-06 Yogesh Rathod Display virtual objects in the event of receiving of augmented reality scanning or photo of real world object from particular location or within geofence and recognition of real world object
CN109078327A (zh) * 2018-08-28 2018-12-25 百度在线网络技术（北京）有限公司 基于ar的游戏实现方法和设备
CN109865289A (zh) * 2019-01-15 2019-06-11 特斯联（北京）科技有限公司 一种基于增强现实技术的实景环境娱乐系统及其方法
US20190196690A1 (en) * 2017-06-23 2019-06-27 Zyetric Virtual Reality Limited First-person role playing interactive augmented reality
CN111359200A (zh) * 2020-02-26 2020-07-03 网易（杭州）网络有限公司 基于增强现实的游戏交互方法和装置
CN113952712A (zh) * 2021-09-26 2022-01-21 杭州易现先进科技有限公司 一种基于ar技术实现密室游戏的方法、系统、装置和介质

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120122570A1 (en) * 2010-11-16 2012-05-17 David Michael Baronoff Augmented reality gaming experience
CN108421252A (zh) * 2017-02-14 2018-08-21 深圳梦境视觉智能科技有限公司 一种基于ar设备的游戏实现方法和ar设备
US20190196690A1 (en) * 2017-06-23 2019-06-27 Zyetric Virtual Reality Limited First-person role playing interactive augmented reality
CN107596683A (zh) * 2017-09-25 2018-01-19 晋江市博感电子科技有限公司 基于增强现实的童车虚拟游乐方法、装置及系统
US20180349703A1 (en) * 2018-07-27 2018-12-06 Yogesh Rathod Display virtual objects in the event of receiving of augmented reality scanning or photo of real world object from particular location or within geofence and recognition of real world object
CN109078327A (zh) * 2018-08-28 2018-12-25 百度在线网络技术（北京）有限公司 基于ar的游戏实现方法和设备
CN109865289A (zh) * 2019-01-15 2019-06-11 特斯联（北京）科技有限公司 一种基于增强现实技术的实景环境娱乐系统及其方法
CN111359200A (zh) * 2020-02-26 2020-07-03 网易（杭州）网络有限公司 基于增强现实的游戏交互方法和装置
CN113952712A (zh) * 2021-09-26 2022-01-21 杭州易现先进科技有限公司 一种基于ar技术实现密室游戏的方法、系统、装置和介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118247700A (zh) * 2024-03-19 2024-06-25 西安隆腾科技文化有限公司 一种多媒体信息交互方法及系统
CN118247700B (zh) * 2024-03-19 2024-09-20 西安隆腾科技文化有限公司 一种多媒体信息交互方法及系统
CN118433435A (zh) * 2024-06-27 2024-08-02 广州市锐星信息科技有限公司 一种示教直播系统

Also Published As

Publication number Publication date
CN116943191A (zh) 2023-10-27

Similar Documents

Publication Publication Date Title
WO2023201937A1 (zh) 基于故事场景的人机互动方法、装置、设备及介质
US20230330541A1 (en) Method and apparatus for man-machine interaction based on story scene, device and medium
US20100060662A1 (en) Visual identifiers for virtual world avatars
WO2022252866A1 (zh) 一种互动处理方法、装置、终端及介质
CN107533727A (zh) 全息交互式零售系统
WO2013120851A1 (en) Method for sharing emotions through the creation of three-dimensional avatars and their interaction through a cloud-based platform
KR20240112969A (ko) 공유 증강 현실 박스 열기 경험
Nguyen et al. Real-time 3D human capture system for mixed-reality art and entertainment
US20240013463A1 (en) Applying animated 3d avatar in ar experiences
KR20240125620A (ko) 실시간 상체 의복 교환
JP2023075879A (ja) 情報処理システム、情報処理方法、情報処理プログラム
US20230386147A1 (en) Systems and Methods for Providing Real-Time Composite Video from Multiple Source Devices Featuring Augmented Reality Elements
KR20240125621A (ko) 실시간 모션 및 외관 전달
KR20240128015A (ko) 실시간 의복 교환
CN114026524A (zh) 利用纹理操作的动画化人脸
US20240346762A1 (en) Animatable garment extraction through volumetric reconstruction
US20240355019A1 (en) Product image generation based on diffusion model
US20240346763A1 (en) Body mesh reconstruction from rgb image
Ma et al. Embodied Cognition Guides Virtual-Real Interaction Design to Help Yicheng Flower Drum Intangible Cultural Heritage Dissemination
US11983819B2 (en) Methods and systems for deforming a 3D body model based on a 2D image of an adorned subject
US20240282015A1 (en) Augmented reality experience with lighting adjustment
US12047337B1 (en) Generating media content items during user interaction
US20240331323A1 (en) Systems, methods, and computer-readable media for 3d asset compiler
US20240303926A1 (en) Hand surface normal estimation
US20240346775A1 (en) Stationary extended reality device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22938162

Country of ref document: EP

Kind code of ref document: A1