WO2020238741A1 - 图像处理方法、相关设备及计算机存储介质 - Google Patents

图像处理方法、相关设备及计算机存储介质 Download PDF

Info

Publication number
WO2020238741A1
WO2020238741A1 PCT/CN2020/091503 CN2020091503W WO2020238741A1 WO 2020238741 A1 WO2020238741 A1 WO 2020238741A1 CN 2020091503 W CN2020091503 W CN 2020091503W WO 2020238741 A1 WO2020238741 A1 WO 2020238741A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
head
electronic device
mounted electronic
moment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/091503
Other languages
English (en)
French (fr)
Inventor
梁天鹰
赖武军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of WO2020238741A1 publication Critical patent/WO2020238741A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems

Definitions

  • the present invention relates to the field of communication technology, in particular to image processing methods, related equipment and computer storage media.
  • AR devices can display virtual images for users while viewing real-world scenes. Users can also interact with virtual images to achieve augmented reality effects.
  • head-mounted electronic devices use the framework shown in FIG. 1 to realize the display of composite images.
  • the head-mounted electronic device collects real images in the real world through the camera, and processes the real image and the current posture information of the head-mounted electronic device through the algorithm processing module to generate virtual images.
  • the real image and the virtual image are synthesized into a composite image to be displayed, and then the composite image is projected onto the display of the head-mounted electronic device for the user to view.
  • the head-mounted electronic device consumes a relatively long time delay from the collection of real images to the display of composite images, and the user experience is not good.
  • the embodiment of the invention discloses an image processing method, related equipment and a computer storage medium, which can solve the problems of large time delay in the existing image display.
  • an embodiment of the present invention discloses an image processing method applied to a head-mounted electronic device.
  • the method includes: the head-mounted electronic device acquires a real image collected by a camera at a first moment, and the real image can be
  • the viewing angle of the real image is usually larger than the viewing angle of the image to be displayed by the head-mounted electronic device.
  • the viewing angle of the image collected by the camera is larger than the viewing angle of the image displayed by the head-mounted electronic device.
  • the head-mounted electronic device can obtain its own posture information at the first moment, which is used to reflect the movement posture of the head-mounted electronic device at the first moment, and the posture information includes but is not limited to the head-mounted electronic device The attitude angle (such as depression angle, elevation angle, etc.), position, movement speed, movement angular speed, or other attitude-related information.
  • the head-mounted electronic device can calculate the re-projection matrix according to the posture information at the first moment, and then use the re-projection matrix to process the real image to obtain the image to be displayed by the head-mounted electronic device at the second moment.
  • the re-projection matrix can be used to reflect the mapping relationship between the image that the head-mounted electronic device needs to display at the first moment and the image that needs to be displayed at the second moment, which is later than the first moment.
  • the problems of large time delay and poor user experience in the prior art can be solved, thereby improving the efficiency of image display and improving user experience.
  • the head-mounted electronic device can predict the posture information at the second time according to the posture information at the first time, and then calculate the posture information at the first time and the second time respectively. Obtain the reprojection matrix.
  • the posture information of the head-mounted electronic device may include movement speed and movement angular velocity.
  • the head-mounted electronic device can calculate the relative rotation angle and relative translation vector of the head-mounted electronic device at the second moment relative to the first moment according to the movement speed and the movement angular velocity at the first moment. Further, the head-mounted electronic device calculates and obtains the reprojection matrix according to the calculated relative rotation angle and relative translation vector.
  • the head-mounted electronic device may calculate a rotation matrix according to the relative rotation angle and the rotation direction of the relative rotation angle, and the rotation matrix is used to indicate that the second moment is relative to the first moment. In terms of the relative rotation of the head-mounted electronic device. Further, the head-mounted electronic device calculates and obtains the reprojection matrix according to the relative translation vector and the rotation matrix.
  • the head-mounted electronic device may perform multi-thread processing on real images to obtain processed images. Further, the head-mounted electronic device can use the re-projection matrix to perform image mapping on the processed image to obtain the image to be displayed on the head-mounted electronic device at the second moment.
  • the head-mounted electronic device performs multi-threaded processing of the real image in specific implementations as follows: the head-mounted electronic device preprocesses the real image in the main thread to obtain the preview For the processed real image, the preprocessing includes, but is not limited to, de-distortion processing, correction processing, translation processing, zoom processing, or other image processing. Further, the head-mounted electronic device can obtain a second image based on the pre-processed real image, for example, perform dimensionality reduction processing on the pre-processed real image.
  • the dimensionality reduction processing may include, but is not limited to, channel number reduction processing (such as single channel Image advancement, etc.), resolution reduction processing (such as image downsampling, etc.) or other processing.
  • the head-mounted electronic device may synthesize the preprocessed real image and the respective virtual image of each slave thread to obtain a processed image.
  • the main thread and any slave threads are independent of each other, and any two slave threads also run independently of each other.
  • the main thread and each slave thread support multi-threaded parallel processing, independent of each other and do not affect each other.
  • the second moment is a time point after a preset duration from the first moment, and the preset duration is that the head-mounted electronic device processes the real image for display on the display screen.
  • the length of time is the length of time that the real image is processed to support the image displayed on the head-mounted electronic device.
  • the embodiments of the present invention disclose a head-mounted electronic device including a functional unit for executing the method of the first aspect.
  • an embodiment of the present invention provides yet another head-mounted electronic device, including a memory, a communication interface, and a processor coupled with the memory and the communication interface; the memory is used for storing instructions, and the processor is used for To execute the instructions, the communication interface is used to communicate with other devices (specifically, user equipment, vehicle-mounted devices, or other network devices, such as servers) under the control of the processor; wherein, the processor executes all The method described in the first aspect is executed when the instruction is described.
  • a computer-readable storage medium stores program codes for image processing.
  • the program code includes instructions for executing the method described in the first aspect above.
  • a computer program product including instructions, which when run on a computer, causes the computer to execute the method described in the first aspect.
  • a chip product is provided to implement the foregoing first aspect or the method in any possible implementation manner of the first aspect.
  • FIG. 1 is a schematic diagram of a processing frame of a head-mounted electronic device provided by the prior art.
  • Fig. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present invention.
  • Fig. 3 is a schematic diagram of an image provided by an embodiment of the present invention.
  • Fig. 4 is a schematic diagram of an image operation provided by an embodiment of the present invention.
  • Fig. 5 is a schematic diagram of a multi-thread operation provided by an embodiment of the present invention.
  • FIG. 6 is a schematic structural diagram of a head-mounted electronic device provided by an embodiment of the present invention.
  • FIG. 7 is a schematic structural diagram of another head-mounted electronic device provided by an embodiment of the present invention.
  • the head-mounted electronic device adopts the framework shown in FIG. 1 to realize the composite display of the real image and the virtual image, and there will be a large time delay in the whole process.
  • These delays mainly come from the processing of the camera acquisition and imaging, algorithm processing module and image synthesis module, and the processing between each module or device belongs to serial processing.
  • the processing delay of each module/device is accumulated, which will lead to the overall time. The delay is large and the user experience is reduced.
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present invention.
  • the method shown in Figure 2 includes the following implementation steps:
  • Step S201 The head-mounted electronic device acquires the real image at the first moment collected by the camera.
  • the real image refers to the image that needs to be displayed and collected by the camera in the real world coordinate system.
  • the head-mounted electronic device when the head-mounted electronic device changes its posture, the head-mounted electronic device can collect a real image in the real world at a first moment (such as time T) to predict the second moment ( For example, the image that needs to be displayed at time T+X), to avoid the large time delay between the image displayed by the head-mounted electronic device at time (T+X) and the real image in the real world, which is easy to cause dizziness and affect user perception Experience.
  • a first moment such as time T
  • T+X the image that needs to be displayed at time T+X
  • the head-mounted electronic device can obtain the real image at the first moment collected by the camera.
  • the head-mounted electronic device can call the camera to collect the real world (ie, real world coordinates) at the first moment. Department) real image.
  • the real world coordinate system is a reference coordinate system located in the real world, and the reference coordinate system is usually set manually or customized by the system.
  • the head-mounted electronic device can send a control signal to the camera to call the camera to collect real images at the first moment.
  • the viewing angle of the real image collected by the camera is generally larger than the viewing angle of the image displayed by the head-mounted electronic device.
  • the viewing angle of the real image may be 120°
  • the viewing angle of the image supported by the head-mounted electronic device may be 45°.
  • Fig. 3 for a schematic diagram of image display.
  • the real image collected by the camera supports covering the entire screen, including the puppies, trees, and basketballs in the illustration.
  • the screen supported by the head-mounted electronic device (shown in the figure) is part of the real image, which can be shown in the dashed box in the figure, including a puppy And trees.
  • Step S202 The head-mounted electronic device obtains the posture information at the first moment, and determines the re-projection matrix according to the posture information at the first moment.
  • the re-projection matrix is used to reflect the mapping relationship between the required display images of the head-mounted electronic device at the first moment and the second moment. The second moment is later than the first moment.
  • Step S203 The head-mounted electronic device processes the real image according to the re-projection matrix to obtain the to-be-displayed image of the head-mounted electronic device at the second moment, so that the head-mounted electronic device can display the to-be-displayed image on the display screen. image.
  • the head-mounted electronic device may collect the posture information at the first moment through the motion sensor.
  • the posture information is information used to describe the motion posture of the head-mounted electronic device in the real world, which may include, but is not limited to, the head The pitch angle, attitude angle, movement speed, movement angular speed or other information related to the attitude of the wearable electronic device.
  • the motion sensor includes, but is not limited to, a speed sensor, a gyroscope, a pressure sensor, or other sensors used to measure the attitude of the device, and this application is not limited.
  • the head-mounted electronic device can predict and obtain the posture information at the second moment based on the posture information at the first moment, and calculate the to-be-displayed display that the head-mounted electronic device needs to display at the second moment based on the posture information at the second moment image. Specifically, after obtaining the posture information of the second moment, the head-mounted electronic device can calculate the relative posture information of the second moment relative to the first moment according to the respective posture information of the first moment and the second moment.
  • the information includes, but is not limited to, the relative rotation angle, the rotation direction of the relative rotation angle, the relative translation vector (including displacement and direction), or other information.
  • the head-mounted electronic device can calculate a re-projection matrix based on the relative posture information, and the re-projection matrix is used to indicate the difference between the image that the head-mounted electronic device needs to display at the first moment and the image that needs to be displayed at the second moment. Mapping relations.
  • the head-mounted electronic device can use the re-projection matrix to perform image mapping processing (also called image distortion) on the real image collected at the first moment to obtain the to-be-displayed image that needs to be displayed at the second moment .
  • image mapping processing also called image distortion
  • the head-mounted electronic device may use the reprojection matrix to perform image mapping processing on the processed image to obtain the image to be displayed at the second moment.
  • the processed image is the image obtained after the head-mounted electronic device performs intermediate processing on the real image collected at the first moment.
  • the specific implementation involved in the intermediate processing can be customized by the system, such as image de-distortion, correction, scaling, and synthesis The details of such operations are described below in this application.
  • the first moment in this application may be recorded as time T, and the second moment may be recorded as time (T+X).
  • X represents the time delay from when the head-mounted electronic device collects the real image to when it is processed into the desired display image.
  • the time delay can be obtained by means of maintenance measurement, for example, by using a dedicated time delay test equipment.
  • the time delay X is mainly composed of the following three parts: the time delay of camera acquisition and imaging (that is, the time delay of real image acquisition and imaging), the delay involved in intermediate processing (such as image mapping, etc.), and the display of the head-mounted electronic device The time delay required for the image.
  • the posture information of the head-mounted electronic device includes movement speed and movement angular velocity.
  • the speed of the head-mounted electronic device at the first moment is The angular velocity of motion is
  • the movement speed and the movement angular velocity here represent vectors in a three-dimensional space, and they have directions and magnitudes.
  • the head-mounted electronic device can use the following formula (1) to calculate the relative rotation angle And relative translation vector
  • X represents the time delay between the second time and the first time (may also be referred to as the duration).
  • the head-mounted electronic device may calculate and obtain the rotation matrix according to the relative rotation angle (which may specifically include the magnitude ⁇ of the relative rotation angle and its direction). For example, the relative rotation angle Bring it into the Rodriguez formula to calculate the rotation matrix R, as shown in the following formula (2).
  • n represents the rotation axis, which can be specified by the relative rotation angle Confirm, which means that the angle of ⁇ is rotated around the rotation axis n.
  • T means transpose.
  • the specific calculation method of the reprojection matrix is not limited.
  • the head-mounted electronic device can comprehensively calculate the reprojection matrix according to parameters such as R, L, and the resolution of the real image (or processed image).
  • the reprojection matrix is used to reflect the mapping relationship between the respective display images of the head-mounted electronic devices at the second moment and the first moment.
  • the head-mounted electronic device can use the re-projection matrix to map the processed image (for example, the image after intermediate processing on the real image at the first moment) into the image to be displayed at the second moment, so as The image to be displayed is displayed on the display screen of the head-mounted electronic device for the user to watch.
  • FIG. 4 shows a schematic diagram of an image display effect of a head-mounted electronic device.
  • the image displayed by the head-mounted electronic device at the first time T is the first image
  • the image displayed at the second time (T+X) is the second image (ie, the image to be displayed).
  • the first image includes pictures including puppies and trees
  • the second image includes pictures including data and basketballs.
  • the head-mounted electronic device can collect a real image at time T through a camera. The real image is shown in the figure, including pictures of puppies, trees, and basketballs.
  • the head-mounted electronic device collects its own motion data (ie posture information) at time T through the motion sensor, and then predicts the real image based on the posture information at the first time to obtain the image to be displayed that needs to be displayed at time (T+X) (Ie the second image).
  • T+X time
  • the head-mounted electronic device can process the real image, such as zooming, rotating, shifting, or distorting, to obtain the T+X time according to the predicted viewing angle and picture range (reflected in the reprojection matrix).
  • the second image that needs to be displayed is not repeated here.
  • the head-mounted electronic device may process the real image at the first moment in parallel in a multi-threaded manner to obtain the processed image, so that the subsequent head-mounted electronic device predicts and obtains the image to be displayed at the second moment based on the processed image.
  • the use of a multi-threaded parallel image processing solution can avoid the problem of large time delay consumed by serial processing, reduce the time delay consumed by image intermediate processing, and improve image processing efficiency.
  • the head-mounted electronic device may preprocess the real image at the first moment collected by the camera to obtain the preprocessed real image.
  • the pre-processing may be image processing customized by the system, which may include, but is not limited to, image correction, image scaling, image translation, and image de-distortion.
  • the head-mounted electronic device in one or more slave threads may respectively perform virtual synthesis on the intermediate images to obtain a virtual image corresponding to each slave thread.
  • the intermediate image is obtained based on the pre-processed real image, for example, the pre-processed real image can be obtained by dimensionality reduction processing.
  • the dimensionality reduction processing includes but is not limited to channel number reduction processing, resolution reduction processing, and so on.
  • the head-mounted electronic device can perform single-channel data extraction on the preprocessed real image to convert the RGB image into a black and white (single-channel) image, and furthermore, the black and white image Downsampling is performed to reduce the resolution of the black and white image to obtain a processed intermediate image.
  • the head-mounted electronic device may store the intermediate image in a preset data buffer area for use by other threads.
  • the amount of calculation data of other threads can be reduced, and the calculation complexity and power consumption of other threads can be reduced.
  • any two threads in multithreading are independent of each other and do not affect each other.
  • the main thread and any slave thread, or any two slave threads run independently of each other, which can improve image processing efficiency and avoid serial processing. A lot of time delay consumption.
  • the head-mounted electronic device can use a preset image synthesis algorithm, such as SLAM algorithm, etc., and optionally can also combine data collected by other sensors (for example, a sensor for collecting data related to virtual objects to be synthesized) to perform an intermediate image
  • Virtual synthesis is to add a virtual object displayed for the user in the intermediate image to obtain a synthesized virtual image.
  • the head-mounted electronic device may store the synthesized virtual image in the corresponding data buffer area.
  • the head-mounted electronic device can also convert the preprocessed real image and the virtual image of each slave thread.
  • the images are synthesized to obtain processed images.
  • a locking mechanism can be added between multiple threads to ensure the security of data access.
  • the data can be protected by a locking mechanism to prevent other threads from accessing the data, that is, only the current thread access is run at this time. After the current thread uses and releases the data, you can Notify other threads to access to avoid data tampering, data inconsistency, or data pollution when multiple threads access at the same time.
  • the intermediate image is usually locked using a locking mechanism. At this time, other slave threads or the main thread cannot support access to the intermediate image. Only when the slave thread unlocks the intermediate image and uses the intermediate image, other threads can be notified to access or use the intermediate image.
  • Fig. 5 shows the parallel processing of (X+1) threads to obtain the to-be-displayed image of the head-mounted electronic device at the second time (T+X).
  • thread 0 represents the main thread.
  • the head-mounted electronic device can obtain the Nth frame image collected by the camera at the first moment (time T), and further preprocess the Nth frame image to obtain The Nth frame image after preprocessing.
  • the head-mounted electronic device may perform dimensionality reduction processing on the preprocessed Nth frame image, such as extracting single-channel data, down-sampling, etc., to store the processed image in a data buffer.
  • thread 1 to thread X are all slave threads.
  • the processing flow of each slave thread is similar.
  • the head-mounted electronic device can obtain the N+1 frame image from the data buffer area, and then perform virtual synthesis on the N+1 frame image, as shown in the figure
  • the SLAM algorithm is used for processing to generate corresponding virtual images.
  • the N+1th frame image is the image that the head-mounted electronic device needs to display at the T+1th time, and it may be part or all of the image after the dimensionality reduction process, and it is not limited.
  • the head-mounted electronic device may store the virtual image in the data buffer area for subsequent use.
  • the head-mounted electronic device After the head-mounted electronic device obtains the respective virtual image of each slave thread, it combines the preprocessed Nth frame image of the main thread to synthesize them into a processed image. It is convenient for the subsequent head-mounted electronic device to perform image distortion on the processed image to obtain the to-be-displayed image of the head-mounted electronic device at time (T+X). For details of image distortion, please refer to the relevant description in the foregoing embodiment, which will not be repeated here.
  • the head-mounted electronic device is a type of electronic device, and this application only uses the head-mounted electronic device as an example for illustration, but it does not constitute a limitation, and may also be other electronic devices. Users can wear head-mounted electronic devices to achieve different effects such as virtual reality (VR), AR, and mixed reality (MR).
  • the head-mounted electronic device may be glasses, head-mounted electronic device, goggles, and the like.
  • the electronic device may also be other devices including a display screen, such as an autonomous vehicle including a display screen.
  • FIG. 6 is a schematic structural diagram of a head-mounted electronic device according to an embodiment of the present invention.
  • the head-mounted electronic device 600 shown in FIG. 6 includes a communication unit 602 and a processing unit 604. among them,
  • the communication unit 602 is configured to obtain a real image collected by the camera at the first moment, and the real image is an image that needs to be displayed collected by the camera in the real world coordinate system;
  • the processing unit 604 is configured to determine a re-projection matrix according to the posture information of the head-mounted electronic device at the first moment, and the re-projection matrix is used to reflect that the image to be displayed by the head-mounted electronic device is from the first moment.
  • the processing unit 604 is further configured to process the real image according to the reprojection matrix to obtain the image to be displayed by the head-mounted electronic device at the second moment.
  • the processing unit 604 is specifically configured to predict the posture information at the second time according to the posture information of the head-mounted electronic device at the first time; Calculate the reprojection matrix for the posture information of and the posture information of the second moment.
  • the posture information includes the movement speed and the movement angular velocity of the head-mounted electronic device
  • the processing unit 604 is specifically configured to perform according to the head-mounted electronic device at the first moment.
  • the movement speed and the movement angular velocity, the relative rotation angle and the relative translation vector of the head-mounted electronic device at the second moment relative to the first moment are calculated; according to the relative rotation angle and the relative translation vector, the calculated Describe the reprojection matrix.
  • the processing unit 604 is specifically configured to perform multi-thread processing on the real image to obtain a processed image; and perform image mapping on the processed image according to the reprojection matrix to obtain the processed image.
  • the processing unit 604 is further specifically configured to preprocess the real image in the main thread to obtain the preprocessed real image; in at least one slave thread, the second image is respectively processed Perform virtual synthesis to obtain a virtual image corresponding to each of the slave threads, and the second image is obtained based on the preprocessed real image; and the preprocessed real image and the at least one slave thread are separately The corresponding virtual images are synthesized to obtain the processed image, wherein both the main thread and the at least one slave thread support multi-thread parallel processing.
  • the head-mounted electronic device 600 may further include a storage unit 606 for storing program codes and data of the head-mounted electronic device 600, for example, storing program codes for image processing.
  • the processing unit 604 is configured to call the program code in the storage unit 606 to implement the content in the method embodiment described in FIG. 2 above.
  • each module or unit involved in the head-mounted electronic device 600 of the embodiment of the present invention may be specifically implemented by software programs or hardware.
  • the modules or units involved in the head-mounted electronic device 600 are software modules or software units.
  • the modules or units involved in the head-mounted electronic device 600 can be integrated through dedicated Circuit (application-specific integrated circuit, ASIC) implementation, or programmable logic device (programmable logic device, PLD) implementation, the above-mentioned PLD can be a complex programmable logic device (CPLD), field programmable gate array (field programmable gate array) -programmable gate array (FPGA), general array logic (generic array logic, GAL) or any combination thereof, the present invention is not limited.
  • CPLD complex programmable logic device
  • FPGA field programmable gate array
  • GAL general array logic
  • FIG. 6 is only a possible implementation of the embodiment of the present application.
  • the head-mounted electronic device may also include more or fewer components, which is not limited here.
  • FIG. 7 is a schematic structural diagram of another head-mounted electronic device according to an embodiment of the present invention.
  • the head-mounted electronic device 100 may include a processor 110, a memory 120, a sensor module 130, a microphone 140, buttons 150, an input and output interface 160, a communication module 170, a camera 180, a battery 190, a display screen 1100, and so on.
  • the sensor module 130 may include a motion sensor 131 for detecting and obtaining posture information of the head-mounted electronic device.
  • the sensor module 130 may also include other sensors, such as a sound detector, a proximity light sensor, a distance sensor, a focal length detection optical sensor, an ambient light sensor, an acceleration sensor, and a temperature sensor.
  • the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the head-mounted electronic device 100.
  • the head-mounted electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), video processing unit (VPU) controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processing Neural-network processing unit (NPU), etc.
  • AP application processor
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • VPU video processing unit
  • memory video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • NPU neural network processing Neural-network processing unit
  • the different processing units may be independent devices or integrated in one or more processors.
  • the controller may be the nerve center and command center of the head-mounted electronic device 100.
  • the controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.
  • a memory may also be provided in the processor 110 to store instructions and data.
  • the memory in the processor 110 is a cache memory.
  • the memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
  • the processor 110 may include one or more interfaces.
  • Interfaces may include integrated circuit (I2C) interface, universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general input and output (general -purpose input/output, GPIO) interface, subscriber identity module (SIM) interface, and/or universal serial bus (universal serial bus, USB) interface, serial peripheral interface (serial peripheral interface, SPI) Interface etc.
  • I2C integrated circuit
  • UART universal asynchronous receiver/transmitter
  • MIPI mobile industry processor interface
  • SIM subscriber identity module
  • USB universal serial bus
  • serial peripheral interface serial peripheral interface
  • the I2C interface is a two-way synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL).
  • the processor 110 may include multiple sets of I2C buses.
  • the processor 110 may be respectively coupled to the motion sensor 131, the battery 190, the camera 180, etc. through different I2C bus interfaces.
  • the processor 110 may couple the motion sensor 131 through an I2C interface, so that the processor 110 and the motion sensor 131 communicate through the I2C bus interface to obtain posture information (ie, motion data) of the head-mounted electronic device.
  • the SPI interface can be used for the connection between the processor and the sensor.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a two-way communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • the UART interface is generally used to connect the processor 110 and the communication module 170.
  • the processor 110 communicates with the Bluetooth module in the communication module 170 through the UART interface to implement the Bluetooth function.
  • the MIPI interface can be used to connect the processor 110 with the display screen 1100, the camera 180 and other peripheral devices.
  • the MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc.
  • the processor 110 and the camera 180 communicate through a CSI interface to realize the shooting function of the head-mounted electronic device 100.
  • the processor 110 and the display screen 1100 communicate through a DSI interface to realize the display function of the head-mounted electronic device 100.
  • the GPIO interface can be configured through software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 180, the display screen 1100, the communication module 170, the sensor module 130, the microphone 140, and so on.
  • GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the USB interface 130 is an interface that complies with the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and so on.
  • the USB interface 130 can be used to connect a charger to charge the head-mounted electronic device 100, and can also be used to transfer data between the head-mounted electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through the headphones. This interface can also be used to connect other electronic devices, such as mobile phones.
  • the USB interface can be USB3.0, which is compatible with high-speed display port (DP) signal transmission, and can transmit high-speed video and audio data.
  • DP display port
  • the interface connection relationship between the modules illustrated in the embodiment of the present application is merely illustrative, and does not constitute a structural limitation of the head-mounted electronic device 100.
  • the head-mounted electronic device 100 may also adopt different interface connection modes in the above-mentioned embodiments, or a combination of multiple interface connection modes.
  • the head-mounted electronic device 100 may include a wireless communication function.
  • the communication module 170 may include a wireless communication module and a mobile communication module.
  • the wireless communication function can be realized by an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), a baseband processor (not shown), and the like.
  • the antenna is used to transmit and receive electromagnetic wave signals.
  • the head-mounted electronic device 100 may include multiple antennas, and each antenna may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
  • antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna can be used in combination with a tuning switch.
  • the mobile communication module can provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the head-mounted electronic device 100.
  • the mobile communication module may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.
  • the mobile communication module can receive electromagnetic waves by the antenna, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation.
  • the mobile communication module can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation by the antenna.
  • at least part of the functional modules of the mobile communication module may be provided in the processor 110.
  • at least part of the functional modules of the mobile communication module and at least part of the modules of the processor 110 may be provided in the same device.
  • the modem processor may include a modulator and a demodulator.
  • the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the low-frequency baseband signal is processed by the baseband processor and then passed to the application processor.
  • the application processor outputs sound signals through audio equipment (not limited to speakers, etc.), or displays images or videos through the display screen 1100.
  • the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module or other functional modules.
  • the wireless communication module can provide wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (bluetooth, BT), and global navigation that are used on the head-mounted electronic device 100.
  • WLAN wireless local area networks
  • WiFi wireless fidelity
  • BT Bluetooth
  • global navigation that are used on the head-mounted electronic device 100.
  • Satellite system global navigation satellite system, GNSS
  • frequency modulation frequency modulation, FM
  • near field communication technology near field communication, NFC
  • infrared technology infrared, IR
  • the wireless communication module may be one or more devices integrating at least one communication processing module.
  • the wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110.
  • the wireless communication module may also receive the signal to be sent from the processor 110, perform frequency modulation, amplify, and convert it into electromagnetic waves to radiate through the antenna.
  • the antenna of the head mounted electronic device 100 is coupled with the mobile communication module, so that the head mounted electronic device 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc.
  • the GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite-based augmentation systems
  • the head-mounted electronic device 100 implements a display function through a GPU, a display screen 1100, and an application processor.
  • the GPU is a microprocessor for image processing, connected to the display screen 1100 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations for graphics rendering.
  • the processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
  • the number of display screens 1100 in the head-mounted electronic device 100 may be two, corresponding to the two eyeballs of the user 200 respectively.
  • the content displayed on the two displays can be displayed independently. Different images can be displayed on the two displays to improve the three-dimensional sense of the image.
  • the number of the display screen 1100 in the head-mounted electronic device 100 may also be one to correspond to the two eyeballs of the user 200.
  • the head-mounted electronic device 100 can implement a shooting function through an ISP, a camera 180, a video codec, a GPU, a display screen 1100, and an application processor.
  • the ISP is used to process the data fed back by the camera 180. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transfers the electrical signal to the ISP for processing and is converted into an image visible to the naked eye.
  • ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be provided in the camera 180.
  • the camera 180 is used to capture still images or videos.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other formats.
  • the head-mounted electronic device 100 may include 1 or N cameras 180, and N is a positive integer greater than 1.
  • the camera 180 can be installed on the side of the head-mounted electronic device 100, and can also be installed at a position between two display screens on the head-mounted electronic device 100.
  • the camera 180 is used to capture images and videos within the viewing angle of the user 200 in real time.
  • the head-mounted electronic device 100 generates a virtual image according to the captured real-time image and video, and displays the virtual image on the display screen 1100.
  • the processor 110 may determine the virtual image displayed on the display screen 1100 according to the still image or video image captured by the camera 180, combined with the data (such as brightness, sound, etc.) acquired by the sensor module 130, to achieve superimposition on real world objects Attach a virtual image.
  • the digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals.
  • the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the head mounted electronic device 100 may support one or more video codecs.
  • the head-mounted electronic device 100 can play or record videos in a variety of encoding formats, for example: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
  • MPEG moving picture experts group
  • NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • applications such as intelligent cognition of the head-mounted electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
  • the memory 120 may be used to store computer executable program code, where the executable program code includes instructions.
  • the processor 110 executes various functional applications and data processing of the head-mounted electronic device 100 by running instructions stored in the memory 120.
  • the processor 110 may call instructions stored in the memory 120 to execute all or part of the steps in the method embodiment described in FIG. 2 above.
  • the memory 120 may include a program storage area and a data storage area.
  • the storage program area can store an operating system, at least one application program (such as a sound playback function, an image playback function, etc.) required by at least one function.
  • the data storage area can store data (such as audio data, phone book, etc.) created during the use of the head-mounted electronic device 100.
  • the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.
  • UFS universal flash storage
  • the head-mounted electronic device 100 can implement audio functions through an audio module, a speaker, a microphone 140, a headphone interface, and an application processor. For example, music playback, recording, etc.
  • the audio module is used to convert digital audio information into analog audio signal output, and also used to convert analog audio input into digital audio signal.
  • the audio module can also be used to encode and decode audio signals.
  • the audio module may be provided in the processor 110, or some functional modules of the audio module may be provided in the processor 110.
  • Loudspeakers also called “horns" are used to convert audio electrical signals into sound signals.
  • the head-mounted electronic device 100 can listen to music through a speaker, or listen to a hands-free call.
  • the microphone 140 also called “microphone” or “microphone”, is used to convert sound signals into electrical signals.
  • the head-mounted electronic device 100 may be provided with at least one microphone 140. In some other embodiments, the head-mounted electronic device 100 may be provided with two microphones 140, which can implement a noise reduction function in addition to collecting sound signals. In other embodiments, the head-mounted electronic device 100 may also be provided with three, four or more microphones 140 to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions.
  • the head-mounted electronic device 100 may include a sound detector 132 that can detect and process a voice signal for controlling the portable electronic device.
  • the sound detector may include a microphone 140.
  • the head-mounted electronic device 100 can use the microphone 140 to convert sound into electrical signals.
  • the sound detector 132 can then process the electrical signal and recognize the signal as a command of the head-mounted display system 1300.
  • the processor 110 may be configured to receive a voice signal from the microphone 140. After receiving the voice signal, the processor 110 may run the sound detector 132 to recognize the voice command. For example, when a voice command is received, the head-mounted electronic device 110 can obtain a contact on the stored user contact list, and the head-mounted electronic device 100 can automatically dial the phone number of the contact.
  • the headphone jack is used to connect wired headphones.
  • the headphone interface can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA, CTIA
  • the head-mounted electronic device 100 may include one or more buttons 150, which can control the head-mounted electronic device and provide the user with access to functions on the head-mounted electronic device 100.
  • the key 150 may be in the form of a button, switch, dial, and touch or proximity sensing device (such as a touch sensor).
  • the user 20 can turn on the display screen 1100 of the head-mounted electronic device 100 by pressing a button.
  • the button 190 includes a power button, a volume button, and so on.
  • the button 190 may be a mechanical button. It can also be a touch button.
  • the head mounted electronic device 100 may receive key input, and generate key signal input related to user settings and function control of the head mounted electronic device 100.
  • the head-mounted electronic device 100 may include an input-output interface 160, and the input-output interface 160 may connect other devices to the head-mounted electronic device 100 through appropriate components.
  • Components may include audio/video jacks, data connectors, etc., for example.
  • Sound detectors can detect and process voice signals used to control portable electronic devices.
  • the head-mounted electronic device 100 can implement eye tracking.
  • infrared devices such as infrared transmitters
  • image acquisition devices such as cameras
  • the proximity light sensor may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode.
  • the light emitting diode may be an infrared light emitting diode.
  • the head mounted electronic device 100 emits infrared light to the outside through the light emitting diode.
  • the head mounted electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the head mounted electronic device 100. When insufficient reflected light is detected, the head-mounted electronic device 100 can determine that there is no object near the head-mounted electronic device 100.
  • the head-mounted electronic device 100 may use the proximity light sensor to detect a gesture operation at a specific position of the head-mounted electronic device 100 to achieve the purpose of associating the gesture operation with an operation command.
  • the head mounted electronic device 100 can measure the distance by infrared or laser. In some embodiments, the head-mounted electronic device 100 may use a distance sensor to measure distances to achieve fast focusing.
  • the ambient light sensor is used to sense the brightness of the ambient light.
  • the head mounted electronic device 100 can adaptively adjust the brightness of the display screen 1100 according to the perceived brightness of the ambient light.
  • the ambient light sensor can also be used to automatically adjust the white balance when taking pictures.
  • the acceleration sensor can detect the magnitude of acceleration of the head-mounted electronic device 100 in various directions (generally three-axis). When the head-mounted electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to recognize the posture of head-mounted electronic devices and applied to applications such as pedometers.
  • the temperature sensor is used to detect temperature.
  • the head-mounted electronic device 100 uses the temperature detected by the temperature sensor to execute the temperature processing strategy. For example, when the temperature reported by the temperature sensor exceeds the threshold, the head-mounted electronic device 100 reduces the performance of the processor located near the temperature sensor, so as to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the head-mounted electronic device 100 heats the battery 190 to prevent the head-mounted electronic device 100 from shutting down abnormally due to low temperature. In some other embodiments, when the temperature is lower than another threshold, the head mounted electronic device 100 boosts the output voltage of the battery 190 to avoid abnormal shutdown caused by low temperature.
  • the focal length detection optical sensor is used to detect the focal length of the eyeball of the user 200.
  • the head-mounted electronic device 100 may further include an infrared light source 1200.
  • the focal length detection optical sensor may cooperate with the infrared light source 1200 to detect the focal length of the eyeball of the user 200.
  • the focal length detection optical sensor 132 and the infrared light source 1200 may be arranged on the side of the display screen close to the eyeball.
  • the number of the focal length detecting optical sensor 132 and the infrared light source 1200 may both be two, and each eyeball may correspond to a focal length detecting optical sensor 132 and the infrared light source 1200 for detecting the focal length of the eyeball.
  • the positions and numbers of the focal length detection optical sensor 132, the infrared light source 1200, and the camera 180 on the head mounted electronic device 100 shown in FIG. 7 are only for explaining the embodiment of the present application and should not constitute a limitation.
  • the number of the focal length detecting optical sensor 132 and the infrared light source 1200 may also be one.
  • the number of one focal length detection optical sensor 132 and one infrared light source 1200 can be used to detect the focal length of one eyeball, or detect the focal length of two eyeballs at the same time.
  • the motion sensor 131 may be used to determine the motion posture of the head-mounted electronic device 100.
  • the angular velocity of the head-mounted electronic device 100 around three axes i.e., x, y, and z axes
  • Motion sensors can also be used for navigation, somatosensory game scenes.
  • the motion sensor includes, but is not limited to, a gyroscope sensor, an angular velocity sensor, or a speed sensor.
  • the display screen 1100 is used to display images, videos, etc.
  • the display screen 1100 includes a display panel.
  • the display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active-matrix organic light-emitting diode active-matrix organic light-emitting diode
  • AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the embodiment of the present invention also provides a computer non-transitory storage medium.
  • the computer non-transitory storage medium stores instructions. When it runs on a processor, the method flow shown in FIG. 2 is implemented.
  • the embodiment of the present invention also provides a computer program product.
  • the computer program product runs on a processor, the method flow shown in FIG. 2 is realized.
  • the steps of the method or algorithm described in combination with the disclosure of the embodiment of the present invention may be implemented in a hardware manner, or may be implemented in a manner in which a processor executes software instructions.
  • Software instructions can be composed of corresponding software modules, which can be stored in random access memory (English: Random Access Memory, RAM), flash memory, read-only memory (English: Read Only Memory, ROM), erasable and programmable Read-only memory (English: Erasable Programmable ROM, EPROM), electrically erasable programmable read-only memory (English: EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM, or well-known in the art Any other form of storage medium.
  • An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and can write information to the storage medium.
  • the storage medium may also be an integral part of the processor.
  • the processor and the storage medium may be located in the ASIC.
  • the ASIC may be located in a head-mounted electronic device.
  • the processor and the storage medium may also exist as discrete components in the head-mounted electronic device.
  • the program can be stored in a computer readable storage medium. When executed, it may include the processes of the above-mentioned method embodiments.
  • the aforementioned storage media include: ROM, RAM, magnetic disks or optical disks and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

一种图像处理方法、相关设备及计算机存储介质，其中所述方法包括：头戴式电子设备获取摄像头在第一时刻采集的真实图像（S201），根据第一时刻的姿态信息确定重投影矩阵（S202），然后利用该重投影矩阵对真实图像进行处理，以获得头戴式电子设备在第二时刻的待显示图像（S203）。采用上述方法，能够解决现有图像显示方案中存在的时延较大、用户体验较差等问题。

Description

图像处理方法、相关设备及计算机存储介质
本申请要求在2019年5月24日提交中国国家知识产权局、申请号为201910443434.1的中国专利申请的优先权，发明名称为“图像处理方法、相关设备及计算机存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及通信技术领域，尤其涉及图像处理方法、相关设备及计算机存储介质。
背景技术
头戴电子设备中，增强现实(augmented reality，AR)装置可以在用户观看现实世界场景的同时，为用户显示虚拟图像。用户还可以与虚拟图像进行交互来实现增强现实的效果。
目前，头戴式电子设备采用如图1所示的框架来实现合成图像的显示。如图1，头戴式电子设备通过摄像头采集现实世界中的真实图像，通过算法处理模块对真实图像和头戴式电子设备当前的姿态信息进行处理，以生成虚拟图像，然后通过图像合成模块将真实图像和虚拟图像合成为待显示的合成图像，进而将该合成图像投影到头戴式电子设备的显示器上以供用户查看。
在实践中发现，头戴式电子设备从真实图像的采集到合成图像的显示，整个过程所消耗的时延较大，用户体验不好。
发明内容
本发明实施例公开了图像处理方法、相关设备及计算机存储介质，能够解决现有图像显示中存在的时延较大等问题。
第一方面，本发明实施例公开提供了一种图像处理方法，应用于头戴式电子设备，所述方法包括：头戴式电子设备获取摄像头在第一时刻采集的真实图像，该真实图像可为摄像头在现实世界坐标系下采集的所需显示的图像，该真实图像的视角通常大于头戴式电子设备所需显示的图像的视角。换句话说，摄像头采集图像的视角大于头戴式电子设备所需显示图像的视角。进一步，头戴式电子设备可获取自身在第一时刻的姿态信息，该姿态信息用于反映头戴式电子设备在第一时刻时的运动姿态，该姿态信息包括但不限于头戴式电子设备的姿态角(例如俯角、仰角等)、位置、运动速度、运动角速度、或其他与姿态相关的信息。头戴式电子设备可依据第一时刻的姿态信息，计算获得重投影矩阵，进而利用该重投影矩阵对真实图像进行处理以获得头戴式电子设备在第二时刻的待显示图像。其中，重投影矩阵可用于反映头戴式电子设备在第一时刻所需显示的图像和第二时刻所需显示的图像之间的映射关系，该第二时刻晚于第一时刻。
通过实施本发明实施例，能够解决现有技术中存在时延较大、用户体验不好等问题，从而能提升图像显示的效率，提升用户体验。
结合第一方面，在一些可能的实施例中，头戴式电子设备可根据第一时刻的姿态信息，预测第二时刻的姿态信息，进而根据第一时刻和第二时刻各自的姿态信息，计算获得重投影矩阵。
结合第一方面，在一些可能的实施例中，头戴式电子设备的姿态信息可包括运动速度和 运动角速度。头戴式电子设备可根据第一时刻的运动速度和运动角速度，计算第二时刻相对于第一时刻时头戴式电子设备的相对旋转角度和相对平移向量。进一步，头戴式电子设备根据计算的相对旋转角度和相对平移向量，计算获得重投影矩阵。
结合第一方面，在一些可能的实施例中，头戴式电子设备可根据相对旋转角度和相对旋转角度的旋转方向，计算获得旋转矩阵，该旋转矩阵用于指示第二时刻相对于第一时刻而言，头戴式电子设备的相对旋转状况。进一步，头戴式电子设备根据该相对平移向量和旋转矩阵，计算获得重投影矩阵。
结合第一方面，在一些可能的实施例中，头戴式电子设备可对真实图像进行多线程处理，以获得处理图像。进一步头戴式电子设备可利用重投影矩阵，对该处理图像进行图像映射，以获得头戴式电子设备在第二时刻的待显示图像。
结合第一方面，在一些可能的实施例中，头戴式电子设备对真实图像进行多线程处理的具体实施方式如下：头戴式电子设备在主线程中对真实图像进行预处理，以获得预处理后的真实图像，该预处理包括但不限于去畸变处理、矫正处理、平移处理、缩放处理或其他图像处理等。进一步头戴式电子设备可基于该预处理后的真实图像获得第二图像，例如对预处理后的真实图像进行降维处理，该降维处理可包括但不限于降通道数处理(如单通道图像提前等)、降分辨率处理(例如图像的下采样等)或其他处理等。进而，头戴式电子设备可将预处理后的真实图像和每个从线程各自的虚拟图像进行合成处理，以获得处理图像。
结合第一方面，在一些可能的实施例中，主线程和任意从线程之间相互独立、任意两个从线程之间也相互独立运行。换句话说，主线程和每个从线程均支持多线程并行处理，相互独立、互不影响。
结合第一方面，在一些可能的实施例中，第二时刻为距离第一时刻预设时长后的时间点，该预设时长为头戴式电子设备对真实图像进行处理以在显示屏上显示的时长。换句话说，第二时刻和第一时刻之间的时长为真实图像被处理成支持在头戴式电子设备上显示的图像的时长。
第二方面，本发明实施例公开提供了一种头戴式电子设备，包括用于执行上述第一方面的方法的功能单元。
第三方面，本发明实施例提供了又一种头戴式电子设备，包括存储器、通信接口及与所述存储器和通信接口耦合的处理器；所述存储器用于存储指令，所述处理器用于执行所述指令，所述通信接口用于在所述处理器的控制下与其他设备(具体可为用户设备、车载设备或者其他网络设备，如服务器)进行通信；其中，所述处理器执行所述指令时执行上述第一方面描述的方法。
第四方面，提供了一种计算机可读存储介质，所述计算机可读存储介质存储了用于图像处理的程序代码。所述程序代码包括用于执行上述第一方面描述的方法的指令。
第五方面，提供了一种包括指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述第一方面描述的方法。
第六方面，提供了一种芯片产品，以执行上述第一方面或第一方面的任意可能的实施方式中的方法。
本发明在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍。
图1是现有技术提供的一种头戴式电子设备处理的框架示意图。
图2是本发明实施例提供的一种图像处理方法的流程示意图。
图3是本发明实施例提供的一种图像示意图。
图4是本发明实施例提供的一种图像操作示意图。
图5是本发明实施例提供的一种多线程操作示意图。
图6是本发明实施例提供的一种头戴式电子设备的结构示意图。
图7是本发明实施例提供的另一种头戴式电子设备的结构示意图。
具体实施方式
下面将结合本发明的附图，对本发明实施例中的技术方案进行详细描述。
申请人在提出本申请的过程中发现：头戴式电子设备采用图1所示框架实现真实图像和虚拟图像的合成显示，整个过程中会存在较大的时延。这些时延主要来源于摄像头采集成像、算法处理模块和图像合成模块等模块的处理，且各个模块或器件之间的处理属于串行处理，各模块/器件的处理时延累加，会导致整体时延较大，降低用户体验。
为解决上述问题，本申请特提出一种图像处理方法以及所述方法适用的相关设备。请参见图2，是本发明实施例提供的一种图像处理方法的流程示意图。如图2所示的方法包括如下实施步骤：
步骤S201、头戴式电子设备获取通过摄像头采集的第一时刻的真实图像。该真实图像是指摄像头在现实世界坐标系下采集的所需显示的图像。
本申请中，在头戴式电子设备发生姿态变化时，头戴式电子设备可采集第一时刻(如时刻T)现实世界中的真实图像，以根据该真实图像预测姿态变化后第二时刻(如时刻T+X)所需显示的图像，避免(T+X)时刻头戴式电子设备显示的图像和现实世界中的真实图像之间时延较大，容易让用户造成眩晕，影响用户观感体验。
在实际应用中，头戴式电子设备可获取摄像头采集的第一时刻的真实图像。具体的，当摄像头受控于头戴式电子设备时，例如该摄像头部署在头戴式电子设备内部或外部时，头戴式电子设备可调用摄像头在第一时刻采集现实世界(即现实世界坐标系下)的真实图像。该现实世界坐标系为位于现实世界中的参考坐标系，该参考坐标系通常为人为或系统自定义设置的。当摄像头不受控于头戴式电子设备时，头戴式电子设备可向摄像头发送控制信号，以调用摄像头在第一时刻采集真实图像。
可选地，摄像头采集的真实图像的视角通常大于头戴式电子设备显示图像的视角，例如真实图像的视角可为120°，头戴式电子设备支持显示图像的视角可为45°等。请参见图3示出一种图像显示示意图。如图3所示，摄像头采集的真实图像支持覆盖整个画面，其包括图示中的小狗、树木和篮球等物体。受限于头戴式电子设备的显示视角，头戴式电子设备支持显示的画面(图示为显示图像)为真实图像中的部分画面，其具体可为图示虚线框所示，包括小狗和树木。
步骤S202、头戴式电子设备获取第一时刻的姿态信息，根据第一时刻的姿态信息确定重 投影矩阵。该重投影矩阵用于反映头戴式电子设备在第一时刻和第二时刻各自所需显示图像之间的映射关系。该第二时刻晚于第一时刻。
步骤S203、头戴式电子设备根据该重投影矩阵，对真实图像进行处理，以获得头戴式电子设备在第二时刻的待显示图像，便于头戴式电子设备在显示屏上显示该待显示图像。
本申请中，头戴式电子设备可通过运动传感器采集第一时刻的姿态信息，该姿态信息为用于描述头戴式电子设备在现实世界中的运动姿态的信息，其可包括但不限于头戴式电子设备的俯仰角度、姿态角度、运动速度、运动角速度或其他与姿态相关的信息。该运动传感器包括但不限于速度传感器、陀螺仪、压力传感器或其他用于测量设备姿态的传感器，本申请并不做限定。
进一步头戴式电子设备可根据第一时刻的姿态信息，预测获得第二时刻的姿态信息，并依据第二时刻的姿态信息计算出头戴式电子设备在第二时刻时所需显示的待显示图像。具体的，头戴式电子设备在获得第二时刻的姿态信息后，可根据第一时刻和第二时刻各自的姿态信息，计算出第二时刻相对于第一时刻的相对姿态信息，该相对姿态信息包括但不限于相对旋转角度、相对旋转角度的旋转方向、相对平移向量(包括位移和方向)或其他信息。头戴式电子设备可根据该相对姿态信息计算出重投影矩阵，该重投影矩阵用于指示头戴式电子设备在第一时刻所需显示的图像和第二时刻所需显示的图像之间的映射关系。获得重投影矩阵后，头戴式电子设备可利用该重投影矩阵对第一时刻采集的真实图像进行图像映射处理(也可称为图像扭曲)，以获得第二时刻所需显示的待显示图像。可选地，为提升图像显示效果，头戴式电子设备可利用该重投影矩阵对处理图像进行图像映射处理，以获得第二时刻的待显示图像。该处理图像为头戴式电子设备对第一时刻采集的真实图像进行中间处理后获得的图像，该中间处理涉及的具体实施可为系统自定义设置的，例如图像去畸变、矫正、缩放以及合成等操作处理，具体在本申请下文阐述。
可选地，本申请中第一时刻可记为T时刻，第二时刻可记为(T+X)时刻。其中，X表示头戴式电子设备从采集真实图像开始到处理为所需显示图像所耗的时延。该时延具体可通过维测手段获得，例如采用专用的时延测试设备测试获得。本申请中，时延X主要由以下三部分组成：摄像头采集成像的时延(即真实图像采集成像的时延)、中间处理涉及的时延(例如图像映射等)和头戴式电子设备显示图像所需的时延。
作为一种可能的实施方式，头戴式电子设备的姿态信息包括运动速度和运动角速度。例如，头戴式电子设备在第一时刻的运动速度为
Figure PCTCN2020091503-appb-000001
运动角速度为
Figure PCTCN2020091503-appb-000002
可选地，这里的运动速度和运动角速度表示三维空间的矢量，它们具有方向和大小。相应地，头戴式电子设备可采用如下公式(1)计算获得相对旋转角度
Figure PCTCN2020091503-appb-000003
和相对平移向量
Figure PCTCN2020091503-appb-000004
Figure PCTCN2020091503-appb-000005
其中，X表示第二时刻和第一时刻之间的时延(也可称为时长)。
进一步，头戴式电子设备可根据相对旋转角度(具体可包括相对旋转角度的大小θ和它的方向)计算获得旋转矩阵。例如，将相对旋转角度
Figure PCTCN2020091503-appb-000006
带入罗德里格斯公式中计算获得旋转矩阵R，具体如下公式(2)所示。
R＝cosθ+(1-cosθ)n×n T+sinθ×n^    公式(2)
其中，n表示旋转轴，具体可由相对旋转角度
Figure PCTCN2020091503-appb-000007
确定，表示围绕旋转轴n旋转了θ角度。T表示转置。
头戴式电子设备可根据旋转矩阵R和相对平移向量的平移量L(L＝δ)，计算获得重投影矩阵。关于重投影矩阵的具体计算方式，不做限定。例如，头戴式电子设备可根据R、L以及真实图像(或处理图像)的分辨率等参数，可综合计算获得重投影矩阵。该重投影矩阵用于反映第二时刻相对于第一时刻头戴式电子设备各自的显示图像之间的映射关系。
在计算获得重投影矩阵后，头戴式电子设备可利用该重投影矩阵将处理图像(例如对第一时刻的真实图像进行中间处理后的图像)映射为第二时刻的待显示图像，以将该待显示图像显示在头戴式电子设备的显示屏上，供用户观看。
举例来说，请参见图4示出一种头戴式电子设备显示图像的效果示意图。如图4中，头戴式电子设备在第一时刻T时刻显示的图像为第一图像，在第二时刻(T+X)显示的图像为第二图像(即待显示图像)。其中，第一图像中包括小狗和树木在内的画面，第二图像中包括数据和篮球在内的画面。具体的，头戴式电子设备可通过摄像头采集T时刻的真实图像，该真实图像如图所示，包括小狗、树木和篮球在内的画面。头戴式电子设备通过运动传感器采集T时刻自身的运动数据(即姿态信息)，进而根据第一时刻的姿态信息对真实图像进行预测，以获得(T+X)时刻所需显示的待显示图像(即第二图像)。具体可参见前述实施例，头戴式电子设备可根据预测的视角和画面范围(体现在重投影矩阵)，对真实图像进行处理，例如缩放、旋转、平移或扭曲等，以获得T+X时刻所需显示的第二图像，这里不再赘述。
下面阐述图像中间处理涉及的相关实施例。具体的，头戴式电子设备可采用多线程并行方式对第一时刻的真实图像进行处理，获得处理图像，便于后续头戴式电子设备基于该处理图像预测获得第二时刻的待显示图像。相比于现有技术而言，采用多线程并行的图像处理方案能规避串行处理消耗时延较大的问题，降低图像中间处理消耗的时延，提升图像处理效率。
作为一种可能的实施方式，在主线程中头戴式电子设备可对摄像头采集的第一时刻的真实图像进行预处理，以获得预处理后的真实图像。该预处理可为系统自定义设置的图像处理，其可包括但不限于图像矫正、图像缩放、图像平移以及图像去畸变等处理。进一步，在一个或多个从线程中头戴式电子设备可分别对中间图像进行虚拟合成，以获得每个从线程各自对应的虚拟图像。该中间图像为基于预处理后的真实图像获得的，例如可对预处理后的真实图像进行降维处理获得，该降维处理包括但不限于降通道数处理和降分辨率处理等等。示例性地，假设真实图像为RGB彩色图像，头戴式电子设备可对预处理后的真实图像进行单通道数据提取，以将RGB图像转换为黑白(单通道)图像，进一步还可对黑白图像进行下采样处理以降低黑白图像的分辨率，从而获得处理后的中间图像。
可选地，头戴式电子设备可将中间图像存储至预设的数据缓存区，以便其他线程使用。从而能够减少其他线程的运算数据量，减少其他线程的运算复杂度和功耗。此外，多线程中任意两个线程相互独立、互不影响，例如主线程和任一个从线程，或者任意两个从线程之间相互独立运行，这样可提升图像处理效率，避免串行处理带来的大量时延消耗。
本申请中虚拟合成的具体实施方式，本申请也不做限定。例如，头戴式电子设备可采用 预设图像合成算法，如SLAM算法等，可选地还可结合其他传感器采集的数据(例如待合成的虚拟物体相关数据采集的传感器等)，对中间图像进行虚拟合成，以在中间图像中新增为用户显示的虚拟物体，从而获得合成的虚拟图像。可选地，头戴式电子设备可将合成的虚拟图像存储至相应地数据缓存区。
进一步，头戴式电子设备在获得主线程的预处理后的真实图像和每个从线程的虚拟图像后，头戴式电子设备还可将预处理后的真实图像和每个从线程各自的虚拟图像进行合成，以获得处理图像。
在实际应用中，多线程之间还可增加加锁机制，以保障数据访问的安全性。具体的，当某个线程访问某数据时，可通过加锁机制对该数据进行保护，防止其他线程访问该数据，即此时只运行当前线程访问，在当前线程使用并释放该数据后，可通知其他线程访问，避免多线程同时访问时导致出现数据被篡改、数据不一致或数据污染等问题。例如，在某从线程访问数据存储区中存储的中间图像时，该中间图像通常已利用加锁机制加锁，此时其他从线程或主线程无法支持访问该中间图像。仅当该从线程解锁中间图像，并使用完该中间图像后，可通知其他线程访问或使用该中间图像。
举例来说，以头戴式电子设备的时延为X为例，每份时延运行一个线程为例。如图5示出(X+1)个线程的并行处理以获得第二时刻(T+X)时刻头戴式电子设备的待显示图像。如图5中，线程0表示为主线程，在主线程中头戴式电子设备可获取摄像头在第一时刻(T时刻)采集的第N帧图像，进一步可对第N帧图像进行预处理获得预处理后的第N帧图像。可选地，头戴式电子设备可对预处理后的第N帧图像进行降维处理，例如提取单通道数据、下采样等，以将处理图像存储于数据缓存中。
图中，线程1～线程X均属于从线程。每个从线程的处理流程均类似，以线程1为例头戴式电子设备可从数据缓存区中获取第N+1帧图像，然后对该第N+1帧图像进行虚拟合成，例如图示中采用SLAM算法进行处理，以生成相应地虚拟图像。其中第N+1帧图像为第T+1时刻头戴式电子设备所需显示的图像，其可为降维处理后图像中的部分或全部，不做限定。可选地，头戴式电子设备可将该虚拟图像存储于数据缓存区中，以供后续使用。同理，头戴式电子设备在获得每个从线程各自的虚拟图像后，再结合主线程的预处理后的第N帧图像，将它们合成为处理图像。便于后续头戴式电子设备对该处理图像进行图像扭曲，以获得(T+X)时刻头戴式电子设备的待显示图像。关于图像扭曲具体可参见前述实施例中的相关阐述，这里不再赘述。
需要说明的是，头戴式电子设备属于电子设备的一种，本申请仅以头戴式电子设备为例进行阐述，但并不构成限定，还可以是其他电子设备。用户可以佩戴头戴电子设备实现虚拟现实(virtual reality，VR)、AR、混合现实(mixed reality，MR)等不同效果。例如，头戴电子设备可以是眼镜、头戴电子设备、护目镜等。电子设备还可以是包含显示屏的其他设备，例如包含显示屏的自动驾驶车辆等。
通过实施本发明实施例，能够解决现有图像显示方案中存在的时延较大、用户体验不好等问题。
结合前述图1-图5所述实施例，下面阐述本申请适用的相关设备。请参见图6，是本发明实施例提供的一种头戴式电子设备的结构示意图。如图6所示的头戴式电子设备600包括通信单元602和处理单元604。其中，
通信单元602用于获取摄像头在第一时刻采集的真实图像，所述真实图像为所述摄像头在现实世界坐标系下采集的所需显示的图像；
处理单元604用于根据头戴式电子设备在所述第一时刻的姿态信息，确定重投影矩阵，所述重投影矩阵用于反映所述头戴式电子设备所需显示的图像从所述第一时刻到第二时刻之间的映射关系，所述第二时刻晚于所述第一时刻；
处理单元604还用于根据所述重投影矩阵，对所述真实图像进行处理，以获得所述头戴式电子设备在所述第二时刻的待显示图像。
在一些可能的实施例中，所述处理单元604具体用于根据所述头戴式电子设备在所述第一时刻的姿态信息，预测所述第二时刻的姿态信息；根据所述第一时刻的姿态信息和所述第二时刻的姿态信息，计算重投影矩阵。
在一些可能的实施例中，所述姿态信息包括所述头戴式电子设备的运动速度和运动角速度，所述处理单元604具体用于根据所述头戴式电子设备在所述第一时刻的运动速度和运动角速度，计算所述第二时刻相对于所述第一时刻时所述头戴式电子设备的相对旋转角度和相对平移向量；根据所述相对旋转角度和相对平移向量，计算获得所述重投影矩阵。
在一些可能的实施例中，所述处理单元604具体用于对所述真实图像进行多线程处理，以获得处理图像；根据所述重投影矩阵，对所述处理图像进行图像映射，以获得所述头戴式电子设备在所述第二时刻的待显示图像。
在一些可能的实施例中，所述处理单元604还具体用于在主线程中对所述真实图像进行预处理，以获得预处理后的真实图像；在至少一个从线程中分别对第二图像进行虚拟合成，以获得每个所述从线程对应的虚拟图像，所述第二图像基于所述预处理后的真实图像获得；将所述预处理后的真实图像和所述至少一个从线程各自对应的虚拟图像进行合成，以获得所述处理图像，其中所述主线程和所述至少一个从线程都支持多线程并行处理。
可选地，头戴式电子设备600还可包括存储单元606，该存储单元606用于存储头戴式电子设备600的程序代码和数据，例如存储用于图像处理的程序代码。处理单元604用于调用该存储单元606中的程序代码以实现如上图2所述方法实施例中的内容。
在实际应用中，本发明实施例的头戴式电子设备600中涉及的各模块或单元具体可通过软件程序或硬件实现。当由软件程序实现时，头戴式电子设备600中涉及的各模块或单元均为软件模块或软件单元，当由硬件实现时，头戴式电子设备600涉及的各模块或单元可以通过专用集成电路(application-specific integrated circuit，ASIC)实现，或可编程逻辑器件(programmable logic device，PLD)实现，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)，现场可编程门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic，GAL)或其任意组合，本发明不做限定。
需要说明的，图6仅仅是本申请实施例的一种可能的实现方式，实际应用中，头戴式电子设备还可以包括更多或更少的部件，这里不作限制。关于本发明实施例中未示出或未描述的内容，可参见前述方法实施例中的相关阐述，这里不再赘述。
请参见图7，是本发明实施例提供的另一种头戴式电子设备的结构示意图。当头戴电子设备安装在用户头上时，用户眼睛可以看到头戴电子设备显示屏呈现的图像。如图7所示，头戴电子设备100可以包括处理器110，存储器120，传感器模块130，麦克风140，按键150，输入输出接口160，通信模块170，摄像头180，电池190以及显示屏1100等。其中传感器 模块130可以包括运动传感器131，用于检测获得头戴式电子设备的姿态信息。传感器模块130还可以包含其他传感器，例如声音探测器，接近光传感器，距离传感器，焦距检测光学传感器，环境光传感器，加速度传感器和温度传感器等。
可以理解的是，本申请实施例示意的结构并不构成对头戴电子设备100的具体限定。在本申请另一些实施例中，头戴电子设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，视频处理单元(video processing unit，VPU)控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
其中，控制器可以是头戴电子设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口，串行外设接口(serial peripheral interface，SPI)接口等。
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。处理器110可以通过不同的I2C总线接口分别耦合运动传感器131，电池190，摄像头180等。例如：处理器110可以通过I2C接口耦合运动传感器131，使处理器110与运动传感器131通过I2C总线接口通信，来获取头戴式电子设备的姿态信息(即运动数据)。SPI接口可以用于处理器与传感器之间的连接。
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与通信模块170。例如：处理器110通过UART接口与通信模块170中的蓝牙模块通信，实现蓝牙功能。
MIPI接口可以被用于连接处理器110与显示屏1100，摄像头180等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(display serial interface，DSI)等。在一些实施例中，处理器110和摄像头180通过CSI接口通信，实现头戴电子设备100的拍摄功能。处理器110和显示屏1100通过DSI接口通信，实现头戴电子设备100的显示功能。
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头180，显示屏1100，通信模块170，传感器模块130，麦克风140等。GPIO接口还可以被配置为I2C接口，I2S接口，UART接口，MIPI接口等。
USB接口130是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口130可以用于连接充电器为头戴电子设备100充电，也可以用于头戴电子设备100与外围设备之间传输数据。也可以用于连接耳机，通过耳机播放音频。该接口还可以用于连接其他电子设备，例如手机等。USB接口可以是USB3.0，用于兼容高速显示接口(display port，DP)信号传输，可以传输视音频高速数据。
可以理解的是，本申请实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对头戴电子设备100的结构限定。在本申请另一些实施例中，头戴电子设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
另外，头戴电子设备100可以包含无线通信功能。通信模块170可以包含无线通信模块和移动通信模块。无线通信功能可以通过天线(未示出)、移动通信模块(未示出)，调制解调处理器(未示出)以及基带处理器(未示出)等实现。
天线用于发射和接收电磁波信号。头戴电子设备100中可以包含多个天线，每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块可以提供应用在头戴电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块可以由天线接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块还可以对经调制解调处理器调制后的信号放大，经天线转为电磁波辐射出去。在一些实施例中，移动通信模块的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器等)输出声音信号，或通过显示屏1100显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块或其他功能模块设置在同一个器件中。
无线通信模块可以提供应用在头戴电子设备100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块经由天线接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块还可以从处理器110接收待发送的信号，对其进行调频，放大， 经天线转为电磁波辐射出去。
在一些实施例中，头戴电子设备100的天线和移动通信模块耦合，使得头戴电子设备100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。
头戴电子设备100通过GPU，显示屏1100，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏1100和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
头戴电子设备100中显示屏1100的数量可以是两个，分别对应用户200的两个眼球。这两个显示屏上显示的内容可以独立显示。可以在这两个显示屏上显示不同的图像来提高图像的立体感。在一些可能的实施例中，头戴电子设备100中显示屏1100的数量也可以是一个，来对应用户200的两个眼球。
头戴电子设备100可以通过ISP，摄像头180，视频编解码器，GPU，显示屏1100以及应用处理器等实现拍摄功能。
ISP用于处理摄像头180反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头180中。
摄像头180用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，头戴电子设备100可以包括1个或N个摄像头180，N为大于1的正整数。
如图7所示，摄像头180可以安装在头戴电子设备100的侧面，还可以安装在头戴电子设备100上两个显示屏之间的位置。摄像头180用于实时捕捉用户200视角内的图像和视频。头戴电子设备100根据捕获的实时的图像和视频生成虚拟图像，并将虚拟图像通过显示屏1100进行显示。
处理器110可以根据摄像头180捕获的静态图像或视频图像，结合传感器模块130获取的数据(例如亮度、声音等数据)，来确定显示屏1100上显示的虚拟图像，来实现在现实世界物体上叠加上虚拟图像。
其中，数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当头戴电子设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。头戴电子设备100可以支持一种或多种视频编解码器。这样，头戴电子设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现头戴电子设备100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
存储器120可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在存储器120的指令，从而执行头戴电子设备100的各种功能应用以及数据处理。例如，处理器110可调用存储器120中存储的指令执行如上图2所述方法实施例中的所有或部分实施步骤。存储器120可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储头戴电子设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，存储器120可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
头戴电子设备100可以通过音频模块，扬声器，麦克风140，耳机接口，以及应用处理器等实现音频功能。例如音乐播放，录音等。
音频模块用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块还可以用于对音频信号编码和解码。在一些实施例中，音频模块可以设置于处理器110中，或将音频模块的部分功能模块设置于处理器110中。
扬声器，也称“喇叭”，用于将音频电信号转换为声音信号。头戴电子设备100可以通过扬声器收听音乐，或收听免提通话。
麦克风140，也称“话筒”，“传声器”，用于将声音信号转换为电信号。头戴电子设备100可以设置至少一个麦克风140。在另一些实施例中，头戴电子设备100可以设置两个麦克风140，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，头戴电子设备100还可以设置三个，四个或更多麦克风140，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。
在一些实施例中，头戴电子设备100可以包括声音探测器132，该声音探测器132可以检测和处理用于控制便携电子设备的语音信号。例如，声音探测器可以包含麦克风140。便头戴电子设备100可以使用麦克风140将声音转换为电信号。声音探测器132随后可以处理电信号，并将信号识别为头戴式显示系统1300的命令。处理器110可以被配置为从麦克风140接收语音信号。在接收到语音信号后，处理器110可以运行声音探测器132来识别语音命令。例如，当接收到语音指令时，头戴电子设备110可以获取存储的用户联系人列表上的联系人，便头戴电子设备100可以自动拨打该联系人电话号码。
耳机接口用于连接有线耳机。耳机接口可以是USB接口，也可以是3.5mm的开放移动头戴电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
在一些实施例中，头戴电子设备100可以包括一个或多个按键150，这些按键可以控制头戴电子设备，为用户提供访问头戴电子设备100上的功能。按键150的形式可以是按钮、开关、刻度盘和触摸或近触摸传感设备(如触摸传感器)。具体的，例如，用户20可以通过 按下按钮来打开头戴电子设备100的显示屏1100。按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。头戴电子设备100可以接收按键输入，产生与头戴电子设备100的用户设置以及功能控制有关的键信号输入。
在一些实施例中，头戴电子设备100可以包括输入输出接口160，输入输出接口160可以通过合适的组件将其他装置连接到头戴电子设备100。组件例如可以包括音频/视频插孔，数据连接器等。
声音探测器可以检测和处理用于控制便携电子设备的语音信号。
在一些实施例中，头戴电子设备100可以实现眼球追踪(eye tracking)。具体的，可以利用红外设备(如红外发射器)和图像采集设备(如摄像头)来检测眼球注视方向。
接近光传感器可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。头戴电子设备100通过发光二极管向外发射红外光。头戴电子设备100使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定头戴电子设备100附近有物体。当检测到不充分的反射光时，头戴电子设备100可以确定头戴电子设备100附近没有物体。头戴电子设备100可以利用接近光传感器检测头戴电子设备100特定位置的手势操作，以实现手势操作与操作命令相关联的目的。
距离传感器，用于测量距离。头戴电子设备100可以通过红外或激光测量距离。在一些实施例中，头戴电子设备100可以利用距离传感器测距以实现快速对焦。
环境光传感器用于感知环境光亮度。头戴电子设备100可以根据感知的环境光亮度自适应调节显示屏1100亮度。环境光传感器也可用于拍照时自动调节白平衡。
加速度传感器可检测头戴电子设备100在各个方向上(一般为三轴)加速度的大小。当头戴电子设备100静止时可检测出重力的大小及方向。还可以用于识别头戴电子设备姿态，应用于计步器等应用。
温度传感器用于检测温度。在一些实施例中，头戴电子设备100利用温度传感器检测的温度，执行温度处理策略。例如，当温度传感器上报的温度超过阈值，头戴电子设备100执行降低位于温度传感器附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，头戴电子设备100对电池190加热，以避免低温导致头戴电子设备100异常关机。在其他一些实施例中，当温度低于又一阈值时，头戴电子设备100对电池190的输出电压执行升压，以避免低温导致的异常关机。
焦距检测光学传感器用于检测用户200的眼球的焦距。在一些实施例中，如图7所示，头戴电子设备100还可以包含红外线光源1200。焦距检测光学传感器可以是和红外线光源1200配合检测用户200的眼球的焦距。
如图7所示，焦距检测光学传感器132和红外线光源1200可以设置在显示屏靠近眼球的一侧。焦距检测光学传感器132和红外线光源1200的数量可以均为两个，每个眼球可以对应一个焦距检测光学传感器132和红外线光源1200，用于检测眼球聚焦的焦距。
可以理解的，图7中示出的焦距检测光学传感器132、红外线光源1200和摄像头180的在头戴电子设备100上的位置和数量仅用于解释本申请实施例，不应构成限定。焦距检测光学传感器132和红外线光源1200的数量还可以是一个。一个焦距检测光学传感器132和一个红外线光源1200的数量可以用于检测一个眼球聚焦的焦距，或者同时检测两个眼球聚焦的焦距。
运动传感器131可以用于确定头戴电子设备100的运动姿态。在一些实施例中，可以通 过运动传感器确定头戴电子设备100围绕三个轴(即，x，y和z轴)的角速度。运动传感器还可以用于导航，体感游戏场景。该运动传感器包括但不限于陀螺仪传感器、角速度传感器、或速度传感器等。
显示屏1100用于显示图像，视频等。显示屏1100包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。
本发明实施例还提供一种计算机非瞬态存储介质，所述计算机非瞬态存储介质中存储有指令，当其在处理器上运行时，图2所示的方法流程得以实现。
本发明实施例还提供一种计算机程序产品，当所述计算机程序产品在处理器上运行时，图2所示的方法流程得以实现。
结合本发明实施例公开内容所描述的方法或者算法的步骤可以硬件的方式来实现，也可以是由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(英文：Random Access Memory，RAM)、闪存、只读存储器(英文：Read Only Memory，ROM)、可擦除可编程只读存储器(英文：Erasable Programmable ROM，EPROM)、电可擦可编程只读存储器(英文：Electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、只读光盘(CD-ROM)或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于头戴式电子设备中。当然，处理器和存储介质也可以作为分立组件存在于头戴式电子设备中。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。

Claims (15)

  1. 一种图像处理方法，其特征在于，所述方法包括：
    获取摄像头在第一时刻采集的真实图像，所述真实图像为所述摄像头在现实世界坐标系下采集的所需显示的图像；
    根据头戴式电子设备在所述第一时刻的姿态信息，确定重投影矩阵，所述重投影矩阵用于反映所述头戴式电子设备所需显示的图像从所述第一时刻到第二时刻之间的映射关系，所述第二时刻晚于所述第一时刻；
    根据所述重投影矩阵，对所述真实图像进行处理，以获得所述头戴式电子设备在所述第二时刻的待显示图像。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述头戴式电子设备在所述第一时刻的姿态信息，确定重投影矩阵包括：
    根据所述头戴式电子设备在所述第一时刻的姿态信息，预测所述第二时刻的姿态信息；
    根据所述第一时刻的姿态信息和所述第二时刻的姿态信息，计算重投影矩阵。
  3. 根据权利要求1或2所述的方法，其特征在于，所述姿态信息包括所述头戴式电子设备的运动速度和运动角速度，所述根据所述头戴式电子设备在所述第一时刻的姿态信息，确定重投影矩阵包括：
    根据所述头戴式电子设备在所述第一时刻的运动速度和运动角速度，计算所述第二时刻相对于所述第一时刻时所述头戴式电子设备的相对旋转角度和相对平移向量；
    根据所述相对旋转角度和相对平移向量，计算获得所述重投影矩阵。
  4. 根据权利要求3所述的方法，其特征在于，所述根据所述相对旋转角度和相对平移向量，计算获得所述重投影矩阵包括：
    根据所述相对旋转角度和所述相对旋转角度的旋转方向，计算获得旋转矩阵，所述旋转矩阵用于指示所述第二时刻相对于所述第一时刻时所述头戴式电子设备的相对旋转状况；
    根据所述相对平移向量和所述旋转矩阵，计算获得所述重投影矩阵。
  5. 根据权利要求1-4中任一项所述的方法，其特征在于，所述根据所述重投影矩阵，对所述真实图像进行处理，以获得所述头戴式电子设备在所述第二时刻的待显示图像包括：
    对所述真实图像进行多线程处理，以获得处理图像；
    根据所述重投影矩阵，对所述处理图像进行图像映射，以获得所述头戴式电子设备在所述第二时刻的待显示图像。
  6. 根据权利要求5所述的方法，其特征在于，所述对所述真实图像进行多线程处理，以获得处理图像包括：
    在主线程中对所述真实图像进行预处理，以获得预处理后的真实图像；
    在至少一个从线程中分别对中间图像进行虚拟合成，以获得每个所述从线程对应的虚拟图像，所述中间图像基于所述预处理后的真实图像获得；
    将所述预处理后的真实图像和所述至少一个从线程各自对应的虚拟图像进行合成，以获得所述处理图像，其中所述主线程和所述至少一个从线程都支持多线程并行处理。
  7. 根据权利要求6所述的方法，其特征在于，所述中间图像为对所述预处理后的真实图像进行降维处理获得的图像，所述降维处理包括降图像通道处理和/或降图像分辨率处理。
  8. 根据权利要求1-7中任一项所述的方法，其特征在于，所述第二时刻与所述第一时刻 之间的时长为所述真实图像被处理成所述头戴式电子设备所显示的图像的时长。
  9. 一种头戴式电子设备，其特征在于，包括通信单元和处理单元，其中，
    所述通信单元，用于获取头戴式电子设备在第一时刻采集的真实图像，所述真实图像为所述头戴式电子设备在现实世界坐标系下采集的所需显示的图像；
    所述处理单元，用于根据所述头戴式电子设备在所述第一时刻的姿态信息，确定重投影矩阵，所述重投影矩阵用于反映所述头戴式电子设备所需显示的图像从所述第一时刻到第二时刻之间的映射关系，所述第二时刻晚于所述第一时刻；
    所述处理单元，还用于根据所述重投影矩阵，对所述真实图像进行处理，以获得所述头戴式电子设备在所述第二时刻的待显示图像。
  10. 根据权利要求9所述的设备，其特征在于，所述处理单元具体用于：
    根据所述头戴式电子设备在所述第一时刻的姿态信息，预测所述第二时刻的姿态信息；
    根据所述第一时刻的姿态信息和所述第二时刻的姿态信息，计算重投影矩阵。
  11. 根据权利要求9或10所述的设备，其特征在于，所述姿态信息包括所述头戴式电子设备的运动速度和运动角速度，所述处理单元具体用于：
    根据所述头戴式电子设备在所述第一时刻的运动速度和运动角速度，计算所述第二时刻相对于所述第一时刻时所述头戴式电子设备的相对旋转角度和相对平移向量；
    根据所述相对旋转角度和相对平移向量，计算获得所述重投影矩阵。
  12. 根据权利要求9-11所述的设备，其特征在于，所述处理单元具体用于：
    对所述真实图像进行多线程处理，以获得处理图像；
    根据所述重投影矩阵，对所述处理图像进行图像映射，以获得所述头戴式电子设备在所述第二时刻的待显示图像。
  13. 根据权利要求12所述的设备，其特征在于，所述处理单元还具体用于：
    在主线程中对所述真实图像进行预处理，以获得预处理后的真实图像；
    在至少一个从线程中分别对第二图像进行虚拟合成，以获得每个所述从线程对应的虚拟图像，所述第二图像基于所述预处理后的真实图像获得；
    将所述预处理后的真实图像和所述至少一个从线程各自对应的虚拟图像进行合成，以获得所述处理图像，其中所述主线程和所述至少一个从线程都支持多线程并行处理。
  14. 一种头戴式电子设备，其特征在于，包括存储器、通信接口及与所述存储器和通信接口耦合的处理器；所述存储器用于存储指令，所述处理器用于执行所述指令，所述通信接口用于在所述处理器的控制下与其他设备进行通信；其中，所述处理器执行所述指令时执行如上权利要求1-8中任一项所述的方法。
  15. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至8任一项所述方法。
PCT/CN2020/091503 2019-05-24 2020-05-21 图像处理方法、相关设备及计算机存储介质 Ceased WO2020238741A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910443434.1A CN110244840A (zh) 2019-05-24 2019-05-24 图像处理方法、相关设备及计算机存储介质
CN201910443434.1 2019-05-24

Publications (1)

Publication Number Publication Date
WO2020238741A1 true WO2020238741A1 (zh) 2020-12-03

Family

ID=67885069

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/091503 Ceased WO2020238741A1 (zh) 2019-05-24 2020-05-21 图像处理方法、相关设备及计算机存储介质

Country Status (2)

Country Link
CN (1) CN110244840A (zh)
WO (1) WO2020238741A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116112672A (zh) * 2023-02-06 2023-05-12 北京达佳互联信息技术有限公司 视频编码方法、装置、电子设备与存储介质
EP4254139A1 (en) * 2022-03-30 2023-10-04 Holo-Light GmbH Reprojection method for generating reprojected image data, xr projection system, and machine-learning module

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110244840A (zh) * 2019-05-24 2019-09-17 华为技术有限公司 图像处理方法、相关设备及计算机存储介质
CN111027374B (zh) * 2019-10-28 2023-06-30 华为终端有限公司 一种图像识别方法及电子设备
CN112752119B (zh) * 2019-10-31 2023-12-01 中兴通讯股份有限公司 一种时延误差校正方法、终端设备、服务器及存储介质
CN112802131A (zh) * 2019-11-14 2021-05-14 华为技术有限公司 一种图像处理方法及装置
CN111243027B (zh) * 2020-02-28 2023-06-23 京东方科技集团股份有限公司 延时测量方法、装置及系统
CN113589919A (zh) * 2020-04-30 2021-11-02 华为技术有限公司 图像处理的方法和装置
CN111736692B (zh) * 2020-06-01 2023-01-31 Oppo广东移动通信有限公司 显示方法、显示装置、存储介质与头戴式设备
CN114071197B (zh) * 2020-07-30 2024-04-12 华为技术有限公司 投屏数据处理方法和装置
CN112132108A (zh) * 2020-10-09 2020-12-25 安徽江淮汽车集团股份有限公司 地面点云数据的提取方法、装置、设备及存储介质
CN114449251B (zh) * 2020-10-31 2024-01-16 华为技术有限公司 视频透视方法、装置、系统、电子设备及存储介质
CN112380989B (zh) * 2020-11-13 2023-01-24 歌尔科技有限公司 一种头戴显示设备及其数据获取方法、装置和主机
CN112486318B (zh) * 2020-11-26 2024-07-26 北京字跳网络技术有限公司 图像显示方法、装置、可读介质及电子设备
CN117296082A (zh) * 2021-05-20 2023-12-26 华为技术有限公司 一种图像处理方法及装置
CN114640838B (zh) * 2022-03-15 2023-08-25 北京奇艺世纪科技有限公司 画面合成方法、装置、电子设备及可读存储介质
CN119536517B (zh) * 2024-11-07 2025-12-02 甬江实验室 数据处理方法、系统、装置、电子设备、存储介质和产品

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105847785A (zh) * 2016-05-09 2016-08-10 上海乐相科技有限公司 一种图像处理方法、设备和系统
WO2016153603A1 (en) * 2015-03-23 2016-09-29 Intel Corporation Facilitating true three-dimensional virtual representation of real objects using dynamic three-dimensional shapes
CN106502427A (zh) * 2016-12-15 2017-03-15 北京国承万通信息科技有限公司 虚拟现实系统及其场景呈现方法
CN106998409A (zh) * 2017-03-21 2017-08-01 华为技术有限公司 一种图像处理方法、头戴显示器以及渲染设备
CN110244840A (zh) * 2019-05-24 2019-09-17 华为技术有限公司 图像处理方法、相关设备及计算机存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9514571B2 (en) * 2013-07-25 2016-12-06 Microsoft Technology Licensing, Llc Late stage reprojection
CN104202547B (zh) * 2014-08-27 2017-10-10 广东威创视讯科技股份有限公司 投影画面中提取目标物体的方法、投影互动方法及其系统
US10962780B2 (en) * 2015-10-26 2021-03-30 Microsoft Technology Licensing, Llc Remote rendering for virtual images
US9978180B2 (en) * 2016-01-25 2018-05-22 Microsoft Technology Licensing, Llc Frame projection for augmented reality environments
CN105892658B (zh) * 2016-03-30 2019-07-23 华为技术有限公司 基于头戴显示设备预测头部姿态的方法和头戴显示设备
CN109656367A (zh) * 2018-12-24 2019-04-19 深圳超多维科技有限公司 一种应用于vr场景下的图像处理方法、装置及电子设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016153603A1 (en) * 2015-03-23 2016-09-29 Intel Corporation Facilitating true three-dimensional virtual representation of real objects using dynamic three-dimensional shapes
CN105847785A (zh) * 2016-05-09 2016-08-10 上海乐相科技有限公司 一种图像处理方法、设备和系统
CN106502427A (zh) * 2016-12-15 2017-03-15 北京国承万通信息科技有限公司 虚拟现实系统及其场景呈现方法
CN106998409A (zh) * 2017-03-21 2017-08-01 华为技术有限公司 一种图像处理方法、头戴显示器以及渲染设备
CN110244840A (zh) * 2019-05-24 2019-09-17 华为技术有限公司 图像处理方法、相关设备及计算机存储介质

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4254139A1 (en) * 2022-03-30 2023-10-04 Holo-Light GmbH Reprojection method for generating reprojected image data, xr projection system, and machine-learning module
EP4375884A3 (en) * 2022-03-30 2024-07-17 Holo-Light GmbH Reprojection method for generating reprojected image data, xr projection system, and machine-learning module
US12573133B2 (en) 2022-03-30 2026-03-10 Holo-Light Gmbh Reprojection method of generating reprojected image data, XR projection system, and machine-learning circuit
CN116112672A (zh) * 2023-02-06 2023-05-12 北京达佳互联信息技术有限公司 视频编码方法、装置、电子设备与存储介质

Also Published As

Publication number Publication date
CN110244840A (zh) 2019-09-17

Similar Documents

Publication Publication Date Title
WO2020238741A1 (zh) 图像处理方法、相关设备及计算机存储介质
WO2020192458A1 (zh) 一种图像处理的方法及头戴式显示设备
US20220148161A1 (en) Skin detection method and electronic device
US12086957B2 (en) Image bloom processing method and apparatus, and storage medium
WO2020192461A1 (zh) 一种延时摄影的录制方法及电子设备
CN113810601B (zh) 终端的图像处理方法、装置和终端设备
CN111179282A (zh) 图像处理方法、图像处理装置、存储介质与电子设备
CN114494547B (zh) 一种绘制命令处理方法及其相关设备
WO2022262313A1 (zh) 基于画中画的图像处理方法、设备、存储介质和程序产品
CN112954251B (zh) 视频处理方法、视频处理装置、存储介质与电子设备
WO2021052111A1 (zh) 图像处理方法及电子装置
CN115526787B (zh) 视频处理方法和装置
WO2022017261A1 (zh) 图像合成方法和电子设备
TWI818211B (zh) 眼部定位裝置、方法及3d顯示裝置、方法
CN113542613B (zh) 一种用于拍照的装置及方法
CN114489533A (zh) 投屏方法、装置、电子设备及计算机可读存储介质
WO2021057626A1 (zh) 图像处理方法、装置、设备及计算机存储介质
WO2022033344A1 (zh) 视频防抖方法、终端设备和计算机可读存储介质
CN115150542B (zh) 一种视频防抖方法及相关设备
CN115706869B (zh) 终端的图像处理方法、装置和终端设备
CN114257920B (zh) 一种音频播放方法、系统和电子设备
CN115631250B (zh) 图像处理方法与电子设备
US20250166121A1 (en) Image rendering method and apparatus
WO2022179412A1 (zh) 识别方法及电子设备
CN111626931A (zh) 图像处理方法、图像处理装置、存储介质与电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20812957

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20812957

Country of ref document: EP

Kind code of ref document: A1