VR-AR integrated machine terminal true man remote interaction method and the system based on it
Technical field
The present invention relates to the fields for realizing long-distance education virtual reality, and in particular to a kind of VR-AR integrated machine terminal true man
Remote interaction method and system based on it.
Background technique
Chinese patent discloses that application No. is the image processing method of CN201810124075.9 and devices, computer
Device and readable storage medium storing program for executing, wherein if described image data processing method includes: to convert collected YUV coded image
At RGB image include scratch figure target, then take target image from the RGB image;The target image is encoded
Processing obtains image data to be transmitted, wherein the image data to be transmitted includes YUV422 planar format data and alpha logical
Track data.Although this method can reduce the bandwidth for transmitting image data in existing progress virtual scene or 360 degree of panoramic videos,
Be there are the problem of:
It adds, useful AR or VR technology implementation model simulation session expansion of the prior art etc. can using AR or VR technology
Student to be more easier to be brought into learning simulation scene, although above-mentioned image procossing is realized student's realistic operation picture
It is combined together with scene, and general video capture device does not have voice collecting function, and video capture device transmission figure
As data channel due to data volume it is larger, if the transfer function for also undertaking audio stream leads to video capture device transmission channel
Heavy load, and general AR or VR equipment has the function of voice collecting, but the prior art cannot be by the voice of AR or VR equipment
Acquisition function is combined with the image collecting function of video capture device, causes the function of AR or VR equipment that cannot obtain abundant benefit
With.
Summary of the invention
The present invention will provide a kind of VR-AR integrated machine terminal true man remote interaction method and the system based on it, solve
The voice collecting function of AR or VR equipment is combined with the image collecting function of video capture device in the prior art and is caused
The problem of function of AR or VR equipment cannot be fully utilized.
To achieve the above object, present invention employs the following technical solutions:
Present invention firstly provides a kind of VR-AR integrated machine terminal true man's remote interaction methods, include the following steps:
S1, simultaneously cloud processing server obtain coding pretreatment from video capture device and integrated machine terminal equipment respectively
Video flowing and coded audio stream;
S2, cloud processing server handle coding preprocessed video stream and coded audio stream, obtain target video
Stream and audio stream;
S3, cloud processing server synchronize target video stream and audio stream to form audio/video flow;
S4, cloud processing server carry out MIXED COMPRESSION CODING to audio/video flow;
S5, integrated machine terminal equipment from cloud server obtain MIXED COMPRESSION CODING after audio/video flow;
S6, integrated machine terminal equipment decode the audio/video flow after MIXED COMPRESSION CODING, obtain audio/video flow;
S7, integrated machine terminal device plays audio/video flow.
The present invention also provides a kind of systems based on VR-AR integrated machine terminal true man's remote interaction method, comprising: all-in-one machine
Terminal device, video capture device and cloud processing server, integrated machine terminal equipment and video capture device are and cloud
Hold processing server communication connection;
Integrated machine terminal equipment is used to obtain coded audio stream by sampling and coding, receives the mixed of cloud processing server
Audio/video flow after closing compressed encoding, decodes the audio/video flow after MIXED COMPRESSION CODING to obtain audio/video flow, and plays sound view
Frequency flows;
Video capture device for use, pre-process and coding obtain coding preprocessed video stream;
Cloud processing server is used to be handled to obtain target video to coding preprocessed video stream and coded audio stream
Stream and audio stream, target video stream and audio stream are synchronized to form audio/video flow, and carry out mixing compression to audio/video flow
Coding.
Compared with the prior art, the invention has the following beneficial effects:
Integrated machine terminal equipment has voice collecting and video playing dual function, realizes voice and video and separately examines
It surveys, and passes through time alignment after realizing voice and video separate detection, voice and video is superimposed, is avoided because of language
Sound and video separate detection and cause voice and video compare on phenomenon occur, ensure that broadcasting be voice will not on video
The degree of lip-rounding there is the entanglement imagination, improve usage experience, can use the voice collecting function of integrated machine terminal equipment in this way
Get up, and guarantees that voice and video will not entanglement.
Further advantage, target and feature of the invention will be partially reflected by the following instructions, and part will also be by this
The research and practice of invention and be understood by the person skilled in the art.
Specific embodiment
In order to make the present invention realize technological means, creation characteristic, reach purpose and effect more clearly and be apparent to,
The present invention is further elaborated With reference to embodiment:
Embodiment 1:
The invention proposes a kind of VR-AR integrated machine terminal true man's remote interaction methods, include the following steps:
S1, simultaneously cloud processing server obtain coding pretreatment from video capture device and integrated machine terminal equipment respectively
Video flowing and coded audio stream;
S2, cloud processing server handle coding preprocessed video stream and coded audio stream, obtain target video
Stream and audio stream;
S3, cloud processing server synchronize target video stream and audio stream to form audio/video flow;
S4, cloud processing server carry out MIXED COMPRESSION CODING to audio/video flow;
S5, integrated machine terminal equipment from cloud server obtain MIXED COMPRESSION CODING after audio/video flow;
S6, integrated machine terminal equipment decode the audio/video flow after MIXED COMPRESSION CODING, obtain audio/video flow;
S7, integrated machine terminal device plays audio/video flow;
In order to reduce volume of transmitted data, improve data transfer speed, cloud processing server is from video acquisition in step S1
Equipment obtain coding preprocessed video stream the step of include:
The video acquisition unit acquisition video flowing of S111, video capture device, the control unit of video capture device obtain
Video flowing;
S112, video capture device control unit in video picture carry out for the first time buckle as processing obtain pretreatment regard
Frequency flows, and picture is the depth information data by combining depth camera acquisition in preprocessed video stream, show that a human body is big
The block diagram of form is caused, depth information required for picture is buckled comprising subsequent further operation in block diagram and is buckled for the first time as processing;
S113, video capture device control unit using 4:2:2 format-pattern compression coding technology to preprocessed video
Stream only encodes, and obtains coding preprocessed video stream;
S114, video capture device control unit are sent out by the communication unit of video capture device to cloud processing server
Send coding preprocessed video stream.
In order to design the simple acquisition modes of audio stream, cloud processing server is obtained from integrated machine terminal equipment in step S1
The step of taking coded audio stream include:
Microphone on S121, integrated machine terminal equipment acquires audio stream, and the controller in integrated machine terminal equipment obtains
The audio stream of microphone acquisition;
Controller in S122, integrated machine terminal equipment encodes audio stream using AAC Audio compression coding technology,
Obtain coded audio stream;
S123, integrated machine terminal device controller send coded audio stream to cloud processing server by communication module,
Cloud processing server obtains coded audio stream.
The specific steps of step S112 include:
S1121, first in nobody picture, take the image of a frame depth information, the depth as subsequent processing
Figure movement images, are defined as D1;
S1122, it is combined by depth information, the RGB picture of the normal camera lens of every frame in video, one can be obtained based on deep
Spend a width transfer image acquisition of the information in 2-4M, at this time substantially available one it is based on human body and remove the normal of periphery background
Picture image is defined as P1;
S1123, the obtained P1 of S1122 can be rejected into the ground in P1 in conjunction with the D1 progress fusion operation obtained before
Information obtains and substantially eliminates the human normal image of background and ground to the end, is defined as P2, and P2 is the output of preliminary button picture
As a result.
Background human figure is gone in order to further obtain in character image, background human figure will be gone to be put into outdoor scene to facilitate
In be integrated into virtual reality figure, scratch as setting 0 for the transparent channel A of background after processing, then the background pixel that transparent channel is 0
It is changed to the scene image prime number evidence of respective coordinates, fusion obtains virtual reality figure, the time of virtual reality figure and before personage
Image temporal is the same, and the virtual reality figure and audio stream of same time are superimposed according to the time, so that it may be formed virtual
Outdoor scene audio/video flow, by cloud processing server in step S2 to coding preprocessed video stream handle the step of include:
S211, cloud processing server are decoded to obtain preprocessed video stream to coding preprocessed video stream;
S212, cloud processing server carry out cloud later period personage refinement to frame video image every in preprocessed video stream and scratch
As processing, to obtain personage's video flowing of no background, refinement is stingy finer as handling, and allows to identify personage edge,
This is the prior art, and which is not described herein again;
S213, cloud processing server carry out figure picture's effect to frame video image every in personage's video flowing of no background
Dynamic enhancing processing, obtains target video stream.
In order to design simple audio stream coding step, in step S2 cloud processing server to coded audio stream at
The step of reason includes: to be decoded coded audio stream to obtain audio stream.
In order to design video flowing straightforward procedure together synchronous with audio stream, step S3 specifically: by target video
Every frame image is aligned in time with audio each in audio stream in stream, the audio/video flow after being synchronized.
Embodiment 2:
The present embodiment the difference from embodiment 1 is that: the present embodiment provides one kind be based on VR-AR as described in Example 1
The system of integrated machine terminal true man's remote interaction method
A kind of system based on VR-AR integrated machine terminal true man's remote interaction method, comprising: integrated machine terminal equipment, view
Frequency acquisition equipment and cloud processing server, integrated machine terminal equipment and video capture device with cloud processing server
Communication connection;
Integrated machine terminal equipment is used to obtain coded audio stream by sampling and coding, receives the mixed of cloud processing server
Audio/video flow after closing compressed encoding, decodes the audio/video flow after MIXED COMPRESSION CODING to obtain audio/video flow, and plays sound view
Frequency flows;
Video capture device for use, pre-process and coding obtain coding preprocessed video stream;
Cloud processing server is used to be handled to obtain target video to coding preprocessed video stream and coded audio stream
Stream and audio stream, target video stream and audio stream are synchronized to form audio/video flow, and carry out mixing compression to audio/video flow
Coding.
In order to which using the integrated machine terminal equipment with broadcasting and voice collecting function, integrated machine terminal equipment sets for AR
Standby or VR equipment.
Finally, it is stated that the above examples are only used to illustrate the technical scheme of the present invention and are not limiting, although referring to compared with
Good embodiment describes the invention in detail, those skilled in the art should understand that, it can be to skill of the invention
Art scheme is modified or replaced equivalently, and without departing from the objective and range of technical solution of the present invention, should all be covered at this
In the scope of the claims of invention.