WO2019179342A1 - 图像处理方法、图像处理装置、图像处理系统及介质 - Google Patents

图像处理方法、图像处理装置、图像处理系统及介质 Download PDF

Info

Publication number
WO2019179342A1
WO2019179342A1 PCT/CN2019/078014 CN2019078014W WO2019179342A1 WO 2019179342 A1 WO2019179342 A1 WO 2019179342A1 CN 2019078014 W CN2019078014 W CN 2019078014W WO 2019179342 A1 WO2019179342 A1 WO 2019179342A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
pixel
pixel data
image processing
data
Prior art date
Application number
PCT/CN2019/078014
Other languages
English (en)
French (fr)
Inventor
李茜
张�浩
陈丽莉
孙玉坤
苗京花
王雪丰
彭金豹
赵斌
索健文
王立新
范清文
李文宇
陆原介
Original Assignee
京东方科技集团股份有限公司
北京京东方光电科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司, 北京京东方光电科技有限公司 filed Critical 京东方科技集团股份有限公司
Priority to US16/500,538 priority Critical patent/US11127126B2/en
Publication of WO2019179342A1 publication Critical patent/WO2019179342A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/04—Synchronising
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present disclosure relates to the field of image processing, and in particular to an image processing method, an image processing apparatus, an image processing system, and a medium for image display of virtual reality.
  • gaze point rendering technology allows the use of human perception knowledge to save a lot of computational work. In a fully rendered scene, most of the computational work is actually wasted because the human eye can only accept the details of the center of the gaze point. Since the concentration of cones on the retina responsible for viewing color and detail is different, anything that is more than 5° above the human eye's gaze will gradually reduce clarity.
  • an image processing method comprising: determining coordinate values of one or more target pixels in an image, wherein the target pixels are used to segment the image; The coordinate values of the plurality of target pixels perform a mapping operation to obtain first pixel data; the pixel data of the image is acquired as second pixel data; the first pixel data and the second pixel data are synthesized to obtain a composite image.
  • the image processing method further includes transmitting the composite image to a driving unit of the display device, wherein the first pixel data in the composite image is read by the driving unit, and the first The one-pixel data performs an inverse mapping operation to obtain coordinate values of the one or more target pixels, the coordinate values for dividing the image into a plurality of sub-images and respectively driving display.
  • the mapping operation includes determining a transform parameter for the mapping operation; for a coordinate value of each of the one or more target pixels, the coordinate value is based on the transform parameter Transforming into a range of pixel values of the image to obtain first pixel data, wherein the first pixel data includes a quotient obtained by dividing the coordinate value by the transform parameter and dividing the coordinate value by The remainder obtained by transforming the parameters.
  • the image processing method further includes representing the first pixel data using two adjacent pixel values, wherein the adjacent two pixel values include a first bit pixel value and a second bit pixel a value, wherein: a quotient obtained by dividing the coordinate value by the transformation parameter as a first pixel value of the adjacent two pixel values; a remainder obtained by dividing the coordinate value by the transformation parameter As the second pixel value of the two adjacent pixel values.
  • synthesizing the first pixel data and the second pixel data to obtain a composite image comprises: creating a new image; writing the first pixel data to the new image; The second pixel data is written after one pixel of data; the new image after the data is written is used as a composite image.
  • the image processing method further includes determining an optical parameter of a display device to be used to display the image before determining coordinate values of the one or more target pixels; based on optics of the display device The parameter performs an anti-distortion operation on the image.
  • the anti-distortion operation includes: determining an anti-distortion mesh according to an optical parameter of the display device; and generating an image after the anti-distortion operation based on the anti-distortion mesh.
  • the image processing method further includes dividing the image into a plurality of sub-images according to the target pixel, and adjusting a resolution of at least one of the plurality of sub-images such that the plurality of sub-images The resolution of at least one sub-image in the image is higher than the resolution of other sub-images.
  • determining coordinate values of one or more target pixels in the image comprises: acquiring a gaze point coordinate value of the user according to an eyeball tracking algorithm; determining, in the image, corresponding to the gaze point according to the gaze point coordinate value a boundary of a gaze point region of the coordinate value; and a coordinate value of the target pixel is determined based on the boundary pixel of the gaze point region.
  • an image processing apparatus comprising: a determining module configured to determine coordinate values of one or more target pixels in an image, wherein the target pixels are used to segment the image a transformation module configured to perform a mapping operation on the coordinate values of the one or more target pixels determined by the determining module to obtain first pixel data, and an acquiring module configured to acquire pixel data of the image as a second pixel And a synthesis module configured to synthesize the first pixel data obtained by the transformation module and the second pixel data acquired by the acquisition module to obtain a composite image.
  • the image processing apparatus further includes: a transmitting module that transmits the composite image to a driving unit of the display device, wherein the first pixel data in the composite image is read by the driving unit, and Performing an inverse mapping operation on the first pixel data to obtain coordinate values of the one or more target pixels, the coordinate values for dividing the image into a plurality of sub-images and respectively driving display.
  • the transforming operation includes determining a transform parameter for the mapping operation; for a coordinate value of each of the one or more target pixels, the coordinate value is based on the transform parameter Transforming into a range of pixel values of the image to obtain first pixel data, wherein the first pixel data includes a quotient obtained by dividing the coordinate value by the transform parameter and dividing the coordinate value by The remainder obtained by transforming the parameters.
  • the transform module is further configured to represent the first pixel data using two adjacent pixel values, wherein the adjacent two pixel values comprise a first bit pixel value and a second bit pixel a value, wherein: a quotient obtained by dividing the coordinate value by the transformation parameter as a first pixel value of the adjacent two pixel values; a remainder obtained by dividing the coordinate value by the transformation parameter As the second pixel value of the two adjacent pixel values.
  • synthesizing the first pixel data and the second pixel data to obtain a composite image comprises: creating a new image; writing the first pixel data to the new image; The second pixel data is written after one pixel of data; the new image after the data is written is used as a composite image.
  • the image processing apparatus further includes an inverse distortion module configured to determine an optical parameter of a display device to be used to display the image prior to determining coordinate values of the one or more target pixels, And performing an anti-distortion operation on the image based on optical parameters of the display device.
  • an inverse distortion module configured to determine an optical parameter of a display device to be used to display the image prior to determining coordinate values of the one or more target pixels, And performing an anti-distortion operation on the image based on optical parameters of the display device.
  • the anti-distortion operation includes: determining an anti-distortion mesh according to an optical parameter of the display device; and generating an image after the anti-distortion operation based on the anti-distortion mesh.
  • the image processing apparatus further includes: a multi-resolution rendering module configured to divide the image into a plurality of sub-images according to the target pixel, and adjust at least one of the plurality of sub-images The resolution is such that the resolution of at least one of the plurality of sub-images is higher than the resolution of the other sub-images.
  • a multi-resolution rendering module configured to divide the image into a plurality of sub-images according to the target pixel, and adjust at least one of the plurality of sub-images The resolution is such that the resolution of at least one of the plurality of sub-images is higher than the resolution of the other sub-images.
  • the determining module is further configured to: acquire a gaze point coordinate value of the user according to an eyeball tracking algorithm; and determine a boundary of the gaze point region of the image corresponding to the gaze point coordinate value according to the gaze point coordinate value And determining a coordinate value of the target pixel based on the boundary pixel of the attention point area.
  • a computer readable storage medium having stored thereon computer readable instructions for performing an image processing method as described above when the instructions are executed by a computer.
  • an image processing system comprising: an image capture device configured to acquire an image; an image display device including a drive unit; and an image processing device including a receiving module configured to The image capture device receives the image; the determining module is configured to determine coordinate values of one or more target pixels in the image, wherein the target pixel is used to segment the image; and the transform module is configured to be paired The coordinate value of the one or more target pixels performs a mapping operation to obtain first pixel data; the acquisition module is configured to acquire pixel data of the image as second pixel data; and a synthesis module configured to synthesize the first pixel Data and the second pixel data to obtain a composite image; and a transmitting module configured to transmit the composite image to the image display device, wherein the first pixel data in the composite image is read by the driving unit And performing an inverse mapping operation on the first pixel data to obtain the one or more target pixels Coordinate values, the coordinate value for the image into a plurality of sub-images, respectively, and
  • FIG. 1A shows a schematic block diagram of an image processing system in accordance with an embodiment of the present disclosure
  • FIG. 1B shows a schematic block diagram of an image processing apparatus according to an embodiment of the present disclosure
  • FIG. 2 illustrates an exemplary flowchart of an image processing method in accordance with an embodiment of the present disclosure
  • FIG. 3A shows a schematic diagram of raw image data acquired by an image capture device
  • FIG. 3B is a schematic diagram showing image data after anti-distortion processing
  • FIG. 3C shows a schematic diagram of image data after multi-resolution rendering
  • 3D shows a schematic diagram of a composite image including target pixel coordinate data and image data to be displayed
  • 3E is a schematic view showing an image displayed in the image display device
  • FIG. 3F shows a schematic diagram of an image viewed by a user through a display device
  • FIG. 4 shows a flow chart of a mapping operation in accordance with an embodiment of the present disclosure
  • FIG. 5 illustrates a flow chart of determining coordinate values of a target pixel in accordance with an embodiment of the present disclosure.
  • modules in a system in accordance with embodiments of the present disclosure, any number of different modules can be used and executed on a user terminal and/or server.
  • the modules are merely illustrative, and different aspects of the systems and methods may use different modules.
  • the present disclosure provides an image processing method for synchronously transmitting coordinate data of a target pixel in an image and image data by performing a mapping operation on coordinate values of a target pixel in the image.
  • FIG. 1A shows a schematic block diagram of an image processing system in accordance with an embodiment of the present disclosure.
  • image processing system 100 can include image capture device 110, image processing device 120, and image display device 130.
  • image processing system 100 may also be referred to as system 100.
  • Image acquisition device 110 can be configured to acquire image data.
  • image capture device 110 may include one or more devices that may be used for image acquisition, such as cameras, video cameras, and the like.
  • image capture device 110 can be configured to include an image capture device for acquiring images for flat display.
  • the image capture device 110 can also be configured to include at least two image capture devices for acquiring an image for left eye display and an image for right eye display, respectively, to implement display of the stereoscopic image.
  • system 100 can also include gesture collection device 140 configured to acquire a user's viewing gesture.
  • the attitude acquisition device 140 can include a gyroscope, an accelerometer, and/or a geomagnetic sensor, and the like.
  • Data for the posture of the stereoscopic image for the image display device can be obtained by the above-described posture acquiring device 140.
  • the above-described gesture collection device 140 may be disposed in a mobile phone, smart glasses, or smart helmet for viewing images.
  • the image processing device 120 may be configured to receive image data acquired by the image capture device 110 and posture data acquired by the gesture collection device 140, and perform an image processing method described below on the received image data.
  • the image processing device 120 may obtain the posture Euler angles for the image display device by using the posture data collected by the attitude acquisition device 140, and generate the image data generated by the image collection device 110 for use in the posture.
  • the image rendering data of the left and right eyes of the user may be configured to receive image data acquired by the image capture device 110 and posture data acquired by the gesture collection device 140, and perform an image processing method described below on the received image data.
  • the image processing device 120 may obtain the posture Euler angles for the image display device by using the posture data collected by the attitude acquisition device 140, and generate the image data generated by the image collection device 110 for use in the posture.
  • the image rendering data of the left and right eyes of the user may be configured to receive image data acquired by the image capture device 110 and posture data acquired by the gesture collection device 140, and perform an image processing method described below on the received image data.
  • image processing device 120 may perform an anti-distortion operation on the received image data to counteract distortion that would be produced when imaging through the optical element in image display device 130, thereby reducing the likelihood that the user would view the displayed image through the optical element. Observed image distortion.
  • image processing device 120 may perform a multi-resolution rendering operation on the received image or the image subjected to the anti-distortion operation, thereby generating an image to be displayed in image display device 130.
  • the image processing device 120 may acquire the gaze point coordinate value of the user according to the eyeball tracking algorithm, and acquire coordinate values of the target pixel for segmenting the image according to the gaze point coordinate value. Then, the image processing device 120 may divide the image into a plurality of sub-images according to the coordinate values of the target pixel, and adjust a resolution of at least one of the plurality of sub-images so that the sub-image corresponding to the user's gaze point in the image The resolution of the image is higher than the resolution of other sub-images.
  • image processing device 120 may perform a mapping operation on coordinate values of the target pixel described above.
  • the coordinate value of the target pixel may correspond to the first data range, for example, may be (0, 0) to (4320, 4800), that is, the coordinate value may have a data value within the first data range.
  • the image pixel value may correspond to a second data range, such as (0, 255), ie the image pixel value may have a data value that is within the second data range.
  • the mapping operation may implement transforming the data value corresponding to the first data range into a data value corresponding to the second data range, for example, by using a transform parameter determined based on the first data range and the second data range.
  • the data obtained after performing the mapping operation on the coordinate values of the target pixel may be referred to as first pixel data, and the first pixel data corresponds to the second data range of the image pixel values, that is, the image corresponding to the second data range may be implemented
  • the pixel data value represents a target pixel coordinate data value corresponding to the first data range.
  • the image processing device 120 may display the first pixel data and the image to be displayed to the image display device 130 for display.
  • the second pixel data of the image is synthesized together into a composite image, and the composite image is transmitted to the image display device 130 for parsing and display.
  • the first pixel data in the composite image may be separated from the second pixel data by parsing, that is, the first pixel data representing the target pixel coordinate value is distinguished from the second pixel data of the image to be displayed.
  • the image processing device 120 herein can be implemented as one or more dedicated or general purpose computer systems, such as a personal computer, a notebook computer, a tablet computer, a mobile phone, a personal digital assistance (PDA), smart glasses, a smart watch, Smart ring, smart helmet and any smart portable device or wearable device.
  • the image processing device 120 can include a communication port to which is connected a network that implements data communication.
  • Image processing device 120 may also include at least one processor for executing program instructions.
  • Image processing device 120 can include an internal communication bus.
  • Image processing device 120 may include different forms of program storage units and data storage units, such as a hard disk, read only memory (ROM), random access memory (RAM), which can be used to store various processing and/or communication uses. Data files, and possible program instructions executed by the processor.
  • Image processing device 120 may also include an input/output component that supports input/output data flow between image processing device 120 and other components, such as a user interface.
  • the image processing device 120 can also transmit and receive information and data from the network through the communication port.
  • the image processing device 120 can perform data communication and transmission with the image capture device 110, the image display device 130, and the gesture capture device 140.
  • Image display device 130 may be configured to receive image data from image processing device 120.
  • the image display device 130 may receive the composite image generated by the image processing device 120 and parse it to obtain coordinate values of a target pixel for segmenting the image and image data to be displayed, for example, the target pixel
  • the coordinate value can be realized by performing an inverse mapping operation on the first pixel data.
  • image display device 130 can include a drive unit.
  • the driving unit may be configured to parse the image data to be displayed and obtain coordinate values of the target pixel for segmenting the image to be displayed. Based on the coordinate values of the target pixel obtained by the analysis, the driving unit may divide the image to be displayed, and respectively stretch and display the images of the divided different regions.
  • image display device 130 can be a display, smart glasses, smart helmet, or the like.
  • the image capture device 110, the image processing device 120, the image display device 130, and the gesture capture device 140 are presented as separate modules, those skilled in the art will appreciate that the device modules described above may be implemented as separate hardware devices. It can also be integrated into one or more hardware devices. The specific implementation of different hardware devices should not be taken as limiting the scope of the disclosure, as long as the principles of the present disclosure can be implemented.
  • FIG. 1B shows a schematic block diagram of an image processing apparatus according to an embodiment of the present disclosure.
  • image processing device 120 can include a determination module 121 configured to determine coordinate values of target pixels for image segmentation of image rendering data for the user's left and right eyes, respectively. For example, when the user views the image using the virtual reality device, the image processing device 120 may obtain the gaze point coordinate value of the user according to the eyeball tracking algorithm. Based on the acquired user gaze point coordinate values, the image processing device 120 may determine a boundary of the gaze point region corresponding to the gaze point coordinate value in the image according to the gaze point coordinate value. Then, the image processing device 120 may determine coordinate values of the target pixels for image segmentation of the image data to be processed based on the boundary of the attention point region. For example, a gaze point region rendered as a high resolution may be determined based on the gaze point coordinate value.
  • the coordinate value of the boundary point of the attention point area can be used as the coordinate value of the target pixel for dividing the image.
  • the image processing apparatus can also include a transform module 122.
  • the image processing device 120 may acquire the coordinate values of the target pixel, and perform a mapping operation on the coordinate values of the target pixel by the transform module 122 to map the coordinate values of the target pixel to the first pixel data.
  • Performing a mapping operation on the coordinate values of one or more target pixels to obtain the first pixel data refers to converting the coordinate value data of the target pixel into data represented using the image pixel value data.
  • the data range of the coordinate values of the target pixel is between (0, 0) and (4320, 4800), and the data of the image pixel value It is usually between (0,255).
  • the transformation parameter can be some value between (4800/255, 255), for example, the transformation parameter can be 250.
  • the mapping operation may be performed by dividing the coordinate value 4500 by the transform parameter 250 for data operation, and the obtained quotient is 18, and the remainder is 0.
  • the quotient and remainder may be referred to as first pixel data and are between data ranges (0, 255) of image pixel values. That is, it is possible to realize the coordinate data value of the target pixel using the first pixel data in the data range of the image pixel value.
  • the first bit pixel value of the adjacent two pixel values may be used to represent the coordinate-converted high-order pixel gray value
  • the second-bit pixel value represents the coordinate-converted low-order pixel gray value.
  • the high-order pixel gray value may be a quotient of the coordinate value of the target pixel divided by the transform parameter (for example, 18)
  • the low-order pixel gray value may be a remainder obtained by dividing the coordinate value of the target pixel by the transform parameter (for example, 0) ).
  • a plurality of pixel values may also be used to represent one coordinate value.
  • the coordinate values of the target pixel can be mapped to pixel data.
  • the coordinate values of the pixels are in a two-dimensional form
  • the coordinate values of each of the dimensions may be mapped into pixel data by using a mapping method as described above.
  • the same method can be used to convert the coordinate values into pixel data.
  • the transformation parameters need to be determined when converting the coordinate values of the pixels into pixel data.
  • the transformation parameters here can be arbitrarily selected according to actual conditions. For example, it is sufficient that the value of the quotient and the remainder obtained by dividing the coordinate value by the transformation parameter is within the range of the image pixel value.
  • the transform parameter may be any integer between (4800/255, 255). .
  • the transform parameter can be 250, or any other number that satisfies the above conditions and can be represented using pixel grayscale values.
  • the image processing apparatus 120 may further include an acquisition module 123 configured to acquire pixel data of an image to be displayed as the second pixel data.
  • the image processing apparatus 120 may further include a synthesizing module 124 configured to synthesize the first pixel data and the second pixel data to obtain a composite image.
  • the operation of synthesizing the first pixel data and the second pixel data to obtain the composite image may include: creating a new image as a composite image, and writing the first pixel data to the first row of the composite image, and the remaining pixels of the first row Bits are padded with zeros. Thereafter, the second pixel data is written to the composite image.
  • the manner of synthesizing the first pixel data and the second pixel data is not limited to the above examples.
  • the first pixel data and the second pixel data may be synthesized using any combination according to actual conditions.
  • the second pixel data can be written first, followed by the first pixel data.
  • specific data for identification may be written before the first pixel data and/or between the first pixel data and the second pixel data, such that the image display device can identify and distinguish the first pixel data from the second pixel. data.
  • Image processing device 120 may also include a transfer module 125 configured to communicate the composite image described above to image display device 130.
  • image processing device 120 may also include an inverse distortion module 126.
  • the anti-distortion module 126 can perform an anti-distortion operation on the image rendering data for the left and right eyes of the user, respectively, according to the optical parameters of the image display device. That is, according to the optical parameters and screen coordinates of the image display device (such as a virtual reality display device), the anti-distortion mesh is drawn and the texture map is obtained, and the image after the distortion is obtained.
  • the image subjected to the anti-distortion processing is displayed by the corresponding image display device, the user will be able to view an image that is substantially free of distortion.
  • the image processing apparatus 120 may further include a multi-resolution rendering module 127, which may be configured to divide the image after performing the anti-distortion operation into a plurality of sub-images according to the target pixel, and adjust at least one of the plurality of sub-images
  • the resolution of one sub-image is such that the center of the image (or the area corresponding to the user's gaze point) has a higher resolution than the other sub-image resolutions.
  • the center area of the image can be determined as the gaze point area of the user.
  • the gaze point area can be determined as a rectangular area.
  • the image can be divided into nine areas.
  • the multi-resolution rendering module 127 can perform different resolution rendering on the divided nine image regions. For example, an image area corresponding to a gaze point area may have the highest resolution. For other image regions, the corresponding resolution can be determined based on the distance from the gaze point region. For example, the farther the image area is from the gaze point area, the lower the resolution can be set.
  • FIG. 2 illustrates an exemplary flow chart of an image processing method in accordance with an embodiment of the present disclosure.
  • step S201 coordinate values of one or more target pixels in the image are determined, wherein the target pixel is used to segment the image.
  • step S202 a mapping operation is performed on coordinate values of one or more target pixels to obtain first pixel data.
  • step S203 the image is read to obtain second pixel data.
  • step S204 the first pixel data and the second pixel data are synthesized to obtain a composite image.
  • step S205 the composite image is transmitted to the driving unit of the display device, wherein the first pixel data in the composite image is read by the driving unit, and an inverse mapping operation is performed on the first pixel data to obtain one or more target pixels.
  • a coordinate value wherein the coordinate value is used to divide the image into a plurality of sub-images and drive display respectively.
  • performing a mapping operation on the coordinate values of the one or more target pixels to obtain the first pixel data refers to converting the coordinate values of the target pixel into data represented by the pixel data of the image that can be used in the image. .
  • a transformation parameter for the mapping operation is determined, for example, the transformation parameter may be based on a data range of a target pixel and an image pixel value.
  • the range of data is determined.
  • step S402 for the coordinate value of each of the coordinates of the one or more target pixels, the coordinate value is transformed into the pixel value range of the image based on the transformation parameter to obtain the first Pixel data.
  • the first pixel data includes a quotient obtained by dividing the coordinate value by the transform parameter and a remainder obtained by dividing the coordinate value by the transform parameter.
  • the coordinate value of the pixel ranges from (0, 0) to (4320, 4800), and the pixel gray value is generally between (0, 255), and the coordinate value is required. Convert to the interval corresponding to the pixel gray value.
  • the first two pixel values may be used to represent the first pixel data, that is, one adjacent pixel value is used to represent one coordinate value of the target pixel.
  • the first bit pixel value of the adjacent two pixel values represents the coordinate-converted high-order pixel gray value
  • the second-bit pixel value represents the coordinate-converted low-order pixel gray value.
  • the high-order pixel gray value may be a quotient obtained by dividing the coordinate value of the target pixel by the transform parameter
  • the low-order pixel gray value may be a remainder obtained by dividing the coordinate value of the target pixel by the transform parameter.
  • a plurality of pixel values may also be used to represent one coordinate value of the target pixel.
  • the coordinate values of the target pixel can be mapped to pixel data.
  • the coordinate values of the pixels are in a two-dimensional form
  • the coordinate values of each of the dimensions may be mapped into pixel data by using a mapping method as described above.
  • the same method can be used to convert the coordinate values into pixel data.
  • the transformation parameters need to be determined when converting the coordinate values of the pixels into pixel data.
  • the transformation parameters herein can be arbitrarily selected according to the actual situation, and the values of the quotient and remainder obtained by dividing the coordinate value by the transformation parameter fall into the pixel gray value.
  • the scope is fine.
  • the transformation parameter may be any integer between (4800/255, 255).
  • step S204 synthesizing the first pixel data and the second pixel data to obtain the composite image comprises: creating a new image; writing the first pixel data to the first row of the new image; at the first pixel The second pixel data is written after the data; the new image after the data is written is taken as the composite image. For example, the first pixel data is written to the first line of the composite image and the remaining pixel bits of the first line are padded with zeros. Thereafter, the second pixel data is written to the composite image.
  • the manner of synthesizing the first pixel data and the second pixel data is not limited to the above examples.
  • the first pixel data and the second pixel data may be synthesized using any combination according to actual conditions.
  • the second pixel data can be written first, followed by the first pixel data.
  • specific data for identification may be written before the first pixel data and/or between the first pixel data and the second pixel data, such that the image display device can identify and distinguish the first pixel data from the second pixel. data.
  • the image processing method 200 may further include determining an optical parameter of the display device to be used to display the image before determining the coordinate values of the one or more target pixels; performing an anti-distortion on the image based on the optical parameters of the display device operating.
  • the anti-distortion operation may include: determining an anti-distortion mesh according to an optical parameter of the image display device, and attaching a texture of the image to the anti-distortion mesh according to coordinates of the image.
  • the anti-distortion operation can include creating an array of vertices, an array of triangles, and an array of image UV coordinates based on image data acquired by the image acquisition device.
  • the coordinates of the original image data can be coordinate-transformed, and the mesh generated according to the anti-distortion operation is drawn according to the coordinate transformation.
  • the image texture can be pasted on the generated mesh to generate an image after the anti-distortion operation.
  • the image processing method 200 may further include dividing the image after performing the anti-distortion operation into a plurality of sub-images according to the target pixel, and adjusting the resolution of at least one of the plurality of sub-images, for example, , so that the center resolution of the image is higher than the surrounding resolution.
  • the process of multi-resolution rendering will be described hereinafter with reference to FIG. 3C.
  • 3A-3F show schematic diagrams of image processing flows for execution of an image processing system utilizing an embodiment of the present disclosure.
  • FIG. 3A shows a schematic diagram of raw image data acquired by an image capture device.
  • the original image data may be image data directly acquired by the image acquisition device, or may be image data acquired by the image processing device using the aforementioned posture data.
  • FIG. 3B is a schematic diagram showing image data after anti-distortion processing.
  • the anti-distortion operation is determined according to the optical parameters of the corresponding display device. That is, the parameters of the anti-distortion operation may be different for different display devices.
  • the image data is converted into an image having distortion as shown in FIG. 3B.
  • FIG. 3C shows a schematic diagram of image data after multi-resolution rendering.
  • the system 100 can also obtain the coordinate value of the gaze point of the user according to the eyeball tracking algorithm, and perform a multi-resolution rendering operation on the image after the anti-distortion operation according to the gaze point coordinate value.
  • coordinate values of one or more target pixels in the image may be determined according to the determined gaze point coordinate values, and the target pixels are used to segment the image after performing the anti-distortion operation, Used for multi-resolution rendering operations.
  • FIG. 5 illustrates a flow chart of determining coordinate values of a target pixel in accordance with an embodiment of the present disclosure.
  • the gaze point coordinate value of the user is obtained according to the eyeball tracking algorithm.
  • the gaze point of the user can be obtained according to the eyeball tracking algorithm, and the coordinate value of the image center can be used as the gaze point coordinate value of the user. .
  • a boundary of the gaze point region corresponding to the gaze point coordinate value in the image is determined according to the gaze point coordinate value.
  • the gaze point area may be a rectangular area centered on the gaze point.
  • the boundary of the rectangular area is the boundary of the gaze point area.
  • step S503 coordinate values of the target pixel are determined based on the boundary of the attention point area. For example, four vertices of a gaze point area of a rectangle may be determined as the target pixel.
  • the center area of the image can be determined as the gaze area of the user.
  • the gaze area can be determined as a rectangular area.
  • the image on the left side corresponds to the left eye image data
  • the image on the right side corresponds to the right eye image data.
  • the left eye image data may be divided into image areas 1-9
  • the right eye image data may be segmented into image areas 1'-9'.
  • the system 100 can separately render different resolutions of the image regions resulting from the segmentation shown in FIG. 3C.
  • the image area No. 5 located at the center may have the highest resolution.
  • the resolution of the No. 4 image area and the No. 6 image area located on the left and right sides of the image area No. 5 in the vertical direction ie, the connection direction of the No. 1 image area and the No. 7 image area
  • the resolution is the same, and the resolution in the horizontal direction (ie, the wiring direction of the image area No. 4 and the image area No. 6) may be smaller than the resolution of the image area No. 5.
  • it may be half (or 1/3) of the resolution of the image area No.
  • the resolution of the No. 2 image area and the No. 8 image area located on the upper and lower sides of the image area No. 5 in the horizontal direction may be the same as the resolution of the No. 5 image area, and the resolution in the vertical direction may be smaller than
  • the resolution of the image area No. 5, for example, may be half the resolution within the image area of No. 5.
  • the image resolution of the above image area in the horizontal direction and the vertical direction may be Both are smaller than the resolution of the image area No. 5.
  • the image resolutions in the horizontal direction and the vertical direction of the image areas No. 1, No. 3, No. 7, and No. 9 may each be half the resolution in the image area of No. 5.
  • the system 100 generates image data for display via the anti-distortion operation illustrated in FIG. 3B and the multi-resolution rendering operation illustrated in FIG. 3C.
  • FIG. 3D shows a schematic diagram of a composite image including target pixel coordinate data and image data to be displayed.
  • the composite image is one more line of pixel data than the image to be displayed (such as the image in Figure 3C).
  • the composite image may include image data of (m+1) rows*n columns of pixels.
  • an extra row of pixels in the composite image is used to store the coordinate values of the target pixel.
  • the data of the first row of pixels of the composite image includes the first pixel data 301 and the first pixel data 302 obtained by performing a mapping operation on the coordinate values of the target pixel.
  • the first pixel data 301 in the left image in FIG. 3D may correspond to the first pixel in the right image in FIG. 3D by performing a mapping operation on the coordinate values of the target pixel in the left image in FIG. 3C;
  • the data 302 may correspond to being obtained by performing a mapping operation on the coordinate values of the target pixels in the right image in FIG. 3C.
  • image data for display (for example, image data as shown in FIG. 3C) generated by the system 100 can be written from the second row of pixels.
  • FIG. 3E is a schematic diagram showing an image displayed in the image display device.
  • the image display device can read the composite image and analyze the coordinate values of the target pixel and the image data for display therefrom. Based on the coordinate values of the target pixel, the image display device will determine a plurality of sub-regions in the image to be displayed, and respectively stretch and display the images of different resolutions in each sub-region.
  • FIG. 3F shows a schematic diagram of an image viewed by a user through a display device. Since the optical component is imaged in the display device, the image data to be displayed may be distorted after passing through the optical component. Since the original data image has undergone anti-distortion processing as shown in FIG. 3B, when the user views the image through the display device, the distortion generated by the optical element of the display device cancels out the effect of the anti-distortion processing, and the user can watch To an image with no distortion or distortion.
  • the image processing method provided by the embodiment of the present disclosure, synchronous transmission of the coordinate data of the target pixel and the image data to be displayed can be realized, the efficiency of image transmission is improved, and the image can be inversely distorted to reduce the user's viewing image.
  • multi-resolution rendering of the image data can be performed to reduce the transmission pressure of the image data.
  • a computer readable storage medium having stored thereon computer readable instructions that are executable when the instructions are executed by a computer.
  • the image processing method, the image processing device, and the image processing system provided by the present disclosure synchronous transmission of coordinate data and image data can be realized, and efficiency in image transmission is improved.
  • the eyeball tracking algorithm can be used to accurately locate the observation area, and only focus on rendering the area, which can save a lot of work. Due to the limited amount of data received by the hardware, multi-resolution rendering can reduce the amount of data transmitted by the PC to the display and reduce the pressure on data transmission.
  • a computer hardware platform can be utilized as a hardware platform for one or more of the elements described above.
  • the hardware elements, operating systems, and programming languages of such computers are common and it is assumed that those skilled in the art are familiar enough with these techniques to be able to provide the information needed for motion control using the techniques described herein.
  • a computer containing user interface (UI) elements can be used as a personal computer (PC) or other type of workstation or terminal device, and can be used as a server after being properly programmed.
  • PC personal computer
  • server can be used as a server after being properly programmed.
  • a tangible, permanent storage medium may include the memory or memory used by any computer, processor, or similar device or associated module. For example, various semiconductor memories, tape drives, disk drives or anything like that can provide storage functionality for software.
  • All software or parts of it may sometimes communicate over a network, such as the Internet or other communication networks.
  • Such communication can load software from one computer device or processor to another.
  • a hardware platform loaded from a server or host computer of a motion control system to a computer environment, or other computer environment implementing the system, or a similar function related to the information required to provide motion control. Therefore, another medium capable of transmitting software elements can also be used as a physical connection between local devices, such as light waves, electric waves, electromagnetic waves, etc., to be propagated through cables, optical cables, or air.
  • Physical media used for carrier waves such as cables, wireless connections, or fiber optic cables can also be considered as media for carrying software.
  • a "one embodiment,” “an embodiment,” and/or “some embodiments” means a feature, structure, or feature that is associated with at least one embodiment of the present disclosure. Therefore, it should be emphasized and noted that “an embodiment” or “an embodiment” or “an alternative embodiment” that is referred to in this specification two or more times in different positions does not necessarily refer to the same embodiment. . Furthermore, some of the features, structures, or characteristics of one or more embodiments of the present disclosure can be combined as appropriate.
  • aspects of the present disclosure may be illustrated and described by a number of patentable categories or conditions, including any new and useful process, machine, product, or combination of materials, or Any new and useful improvements. Accordingly, various aspects of the present disclosure may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.) or by a combination of hardware and software.
  • the above hardware or software may be referred to as a "data block,” “module,” “engine,” “unit,” “component,” or “system.”
  • aspects of the disclosure may be embodied as a computer product located in one or more computer readable medium(s) including a computer readable program code.
  • the computer readable signal medium may contain a propagated data signal containing a computer program code, e.g., on a baseband or as part of a carrier.
  • the propagated signal may have a variety of manifestations, including electromagnetic forms, optical forms, and the like, or a suitable combination.
  • the computer readable signal medium may be any computer readable medium other than a computer readable storage medium that can be communicated, propagated, or transmitted for use by connection to an instruction execution system, apparatus, or device.
  • Program code located on a computer readable signal medium can be propagated through any suitable medium, including a radio, cable, fiber optic cable, radio frequency signal, or similar medium, or a combination of any of the above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Image Processing (AREA)
  • Geometry (AREA)

Abstract

公开了一种图像处理方法，包括：确定图像中的一个或多个目标像素的坐标值，其中所述目标像素用于对所述图像进行分割(S201)；对所述一个或多个目标像素的坐标值执行映射操作以获得第一像素数据(S202)；读取所述图像以获得第二像素数据(S203)；合成所述第一像素数据以及所述第二像素数据以获得合成图像(S204)。

Description

图像处理方法、图像处理装置、图像处理系统及介质
本申请要求于2018年03月19日提交的中国专利申请201810225841.0的优先权，该中国专利申请的全文通过引用的方式结合于此以作为本申请的一部分。
技术领域
本公开涉及图像处理领域，具体涉及一种用于虚拟现实的图像显示的图像处理方法、图像处理装置、图像处理系统及介质。
背景技术
目前在虚拟现实的应用中对于图像显示分辨率的要求越来越高，注视点渲染技术的发展允许利用人类感知知识来节省大量的计算工作。在一个全渲染的场景中，实际上大部分计算工作是浪费的，因为人眼只能接纳注视点中心的细节。由于负责观察色彩和细节的视网膜上的视锥细胞浓度不同，任何超出人眼注视区5°以上的东西都会逐渐降低清晰度。
发明内容
根据本公开的一方面，提出了一种图像处理方法，包括：确定图像中的一个或多个目标像素的坐标值，其中所述目标像素用于对所述图像进行分割；对所述一个或多个目标像素的坐标值执行映射操作以获得第一像素数据；获取所述图像的像素数据作为第二像素数据；合成所述第一像素数据以及所述第二像素数据以获得合成图像。
在一些实施例中，所述图像处理方法还包括将所述合成图像传送给显示装置的驱动单元，其中通过所述驱动单元读取所述合成图像中的第一像素数据，并对所述第一像素数据执行逆映射操作以获得所述一个或多个目标像素的坐标值，所述坐标值用于将所述图像分割成多个子图像并分别进行驱动显示。
在一些实施例中，所述映射操作包括：确定用于所述映射操作的变换参数；对于所述一个或多个目标像素中的每一个的坐标值，基于所述变换参数 将所述坐标值变换到所述图像的像素值范围之内以获得第一像素数据，其中，所述第一像素数据包括将所述坐标值除以所述变换参数得到的商和将所述坐标值除以所述变换参数得到的余数。
在一些实施例中，所述图像处理方法还包括使用相邻的两个像素值表示所述第一像素数据，其中所述相邻的两个像素值包括第一位像素值和第二位像素值，其中：将所述坐标值除以所述变换参数得到的商作为所述相邻的两个像素值中的第一位像素值；将所述坐标值除以所述变换参数得到的余数作为所述相邻的两个像素值中的第二位像素值。
在一些实施例中，其中合成所述第一像素数据以及所述第二像素数据以获得合成图像包括：建立一个新图像；将所述第一像素数据写入所述新图像；在所述第一像素数据之后写入所述第二像素数据；将写入数据后的所述新图像作为合成图像。
在一些实施例中，所述图像处理方法还包括在确定所述一个或多个目标像素的坐标值之前，确定将用于显示所述图像的显示装置的光学参数；基于所述显示装置的光学参数对所述图像执行反畸变操作。
在一些实施例中，所述反畸变操作包括：根据所述显示装置的光学参数确定反畸变网格；基于所述反畸变网格生成经过反畸变操作后的图像。
在一些实施例中，所述图像处理方法还包括根据所述目标像素将所述图像划分为多个子图像，并调整所述多个子图像中的至少一个子图像的分辨率，使得所述多个子图像中至少一个子图像的分辨率高于其他子图像的分辨率。
在一些实施例中，其中确定图像中的一个或多个目标像素的坐标值包括：根据眼球追踪算法获取用户的注视点坐标值；根据所述注视点坐标值确定所述图像中对应于注视点坐标值的注视点区域的边界；以及基于所述注视点区域的边界像素确定目标像素的坐标值。
根据本公开的另一方面，提供了一种图像处理装置，包括：确定模块，配置成确定图像中的一个或多个目标像素的坐标值，其中所述目标像素用于对所述图像进行分割；变换模块，配置成所述确定模块确定的对所述一个或多个目标像素的坐标值执行映射操作以获得第一像素数据；获取模块，配置成获取所述图像的像素数据作为第二像素数据；以及合成模块，配置成合成所述变换模块获得的所述第一像素数据以及所述获取模块获取的所述第二像素数据以获得合成图像。
在一些实施例中，所述图像处理装置还包括：传送模块，将所述合成图像传送给显示装置的驱动单元，其中通过所述驱动单元读取所述合成图像中的第一像素数据，并对所述第一像素数据执行逆映射操作以获得所述一个或多个目标像素的坐标值，所述坐标值用于将所述图像分割成多个子图像并分别进行驱动显示。
在一些实施例中，所述变换操作包括：确定用于所述映射操作的变换参数；对于所述一个或多个目标像素中的每一个的坐标值，基于所述变换参数将所述坐标值变换到所述图像的像素值范围之内以获得第一像素数据，其中，所述第一像素数据包括将所述坐标值除以所述变换参数得到的商和将所述坐标值除以所述变换参数得到的余数。
在一些实施例中，所述变换模块进一步配置成使用相邻的两个像素值表示所述第一像素数据，其中所述相邻的两个像素值包括第一位像素值和第二位像素值，其中：将所述坐标值除以所述变换参数得到的商作为所述相邻的两个像素值中的第一位像素值；将所述坐标值除以所述变换参数得到的余数作为所述相邻的两个像素值中的第二位像素值。
在一些实施例中，其中合成所述第一像素数据以及所述第二像素数据以获得合成图像包括：建立一个新图像；将所述第一像素数据写入所述新图像；在所述第一像素数据之后写入所述第二像素数据；将写入数据后的所述新图像作为合成图像。
在一些实施例中，所述图像处理装置还包括：反畸变模块，配置成在确定所述一个或多个目标像素的坐标值之前，确定将用于显示所述图像的显示装置的光学参数，并基于所述显示装置的光学参数对所述图像执行反畸变操作。
在一些实施例中，所述反畸变操作包括：根据所述显示装置的光学参数确定反畸变网格；基于所述反畸变网格生成经过反畸变操作后的图像。
在一些实施例中，所述图像处理装置还包括：多分辨率渲染模块，配置成根据所述目标像素将所述图像划分为多个子图像，并调整所述多个子图像中的至少一个子图像的分辨率，使得所述多个子图像中至少一个子图像的分辨率高于其他子图像的分辨率。
在一些实施例中，所述确定模块进一步配置成：根据眼球追踪算法获取用户的注视点坐标值；根据所述注视点坐标值确定所述图像中对应于注视点 坐标值的注视点区域的边界；以及基于所述注视点区域的边界像素确定目标像素的坐标值。
根据本公开的另一方面，还提供了一种计算机可读存储介质，其上存储有计算机可读的指令，当利用计算机执行所述指令时执行如前所述的图像处理方法。
根据本公开的另一方面，还提供了一种图像处理系统，包括：图像采集装置，配置成采集图像；图像显示装置，包括驱动单元；以及图像处理装置，包括，接收模块，配置成从所述图像采集装置接收所述图像；确定模块，配置成确定所述图像中的一个或多个目标像素的坐标值，其中所述目标像素用于对所述图像进行分割；变换模块，配置成对所述一个或多个目标像素的坐标值执行映射操作以获得第一像素数据；获取模块，配置成获取所述图像的像素数据作为第二像素数据；合成模块，配置成合成所述第一像素数据以及所述第二像素数据以获得合成图像；以及传送模块，配置成将所述合成图像传送给所述图像显示装置，其中通过所述驱动单元读取所述合成图像中的第一像素数据，并对所述第一像素数据执行逆映射操作以获得所述一个或多个目标像素的坐标值，所述坐标值用于将所述图像分割成多个子图像并分别进行驱动显示。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员而言，在没有做出创造性劳动的前提下，还可以根据这些附图获得其他的附图。以下附图并未刻意按实际尺寸等比例缩放绘制，重点在于示出本公开的主旨。
图1A示出了根据本公开的实施例的图像处理系统的示意性的框图；
图1B示出了根据本公开的实施例的图像处理装置的示意性的框图；
图2示出了根据本公开的实施例的图像处理方法的示例性的流程图；
图3A示出了是通过图像采集装置获取的原始图像数据的示意图；
图3B示出了经过反畸变处理后的图像数据的示意图；
图3C示出了经过多分辨率渲染后的图像数据的示意图；
图3D示出了包括目标像素坐标数据与要显示的图像数据的合成图像的示意图；
图3E示出了图像显示装置中显示的图像的示意图；
图3F示出了用户通过显示设备观看到的图像的示意图；
图4示出了根据本公开实施例的映射操作的流程图；
图5示出了根据本公开实施例的确定目标像素的坐标值的流程图。
具体实施方式
下面将结合附图对本公开实施例中的技术方案进行清楚、完整地描述，显而易见地，所描述的实施例仅仅是本公开的部分实施例，而不是全部的实施例。基于本公开实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，也属于本公开保护的范围。
本公开中使用的“第一”、“第二”以及类似的词语并不表示任何顺序、数量或者重要性，而只是用来区分不同的组成部分。同样，“包括”或者“包含”等类似的词语意指出现该词前面的元件或者物件涵盖出现在该词后面列举的元件或者物件及其等同，而不排除其他元件或者物件。“连接”或者“相连”等类似的词语并非限定于物理的或者机械的连接，而是可以包括电性的连接，不管是直接的还是间接的。“上”、“下”、“左”、“右”等仅用于表示相对位置关系，当被描述对象的绝对位置改变后，则该相对位置关系也可能相应地改变。
如本公开和权利要求书中所示，除非上下文明确提示例外情形，“一”、“一个”、“一种”和/或“该”等词并非特指单数，也可包括复数。一般说来，术语“包括”与“包含”仅提示包括已明确标识的步骤和元素，而这些步骤和元素不构成一个排它性的罗列，方法或者设备也可能包含其他的步骤或元素。
虽然本公开对根据本公开的实施例的系统中的某些模块做出了各种引用，然而，任何数量的不同模块可以被使用并运行在用户终端和/或服务器上。所述模块仅是说明性的，并且所述系统和方法的不同方面可以使用不同模块。
本公开中使用了流程图用来说明根据本公开的实施例的系统所执行的操作。应当理解的是，前面或下面操作不一定按照顺序来精确地执行。相反，可以按照倒序或同时处理各种步骤。同时，也可以将其他操作添加到这些过程中，或从这些过程移除某一步或数步操作。
本公开提供了一种图像处理方法，通过对图像中目标像素的坐标值进行 映射操作实现图像中目标像素的坐标数据与图像数据同步传输。
图1A示出了根据本公开的实施例的图像处理系统的示意性的框图。如图1A所示，图像处理系统100可以包括图像采集装置110、图像处理装置120以及图像显示装置130。在下文中，图像处理系统100也可以被称作系统100。
图像采集装置110可以配置成用于采集图像数据。在一些实施例中，图像采集装置110可以包括一个或多个可用于图像采集的设备，如照相机、摄像机等。例如，图像采集装置110可以配置成包括一个图像采集设备，用于采集用于平面显示的图像。又例如，图像采集装置110也可以配置成包括至少两个图像采集设备，分别用于采集用于左眼显示的图像和右眼显示的图像，以实现立体图像的显示。
当系统100配置成显示立体图像(如虚拟现实VR、增强现实AR等)时，在一些实施例中，系统100还可以包括配置成采集用户观察姿态的姿态采集装置140。例如，姿态采集装置140可以包括陀螺仪、加速度计和/或地磁传感器等。利用上述姿态采集装置140可以获得用于图像显示装置的、立体图像的姿态的数据。例如，当用户通过手机、智能眼镜、智能头盔等设备观看立体图像时，上述姿态采集装置140可以设置在用于观看图像的手机、智能眼镜或智能头盔中。
图像处理装置120可以配置成用于接收图像采集装置110采集的图像数据以及姿态采集装置140采集的姿态数据，并对接收的图像数据执行下文描述的图像处理方法。在一些实施例中，图像处理装置120可以利用姿态采集装置140采集的姿态数据得到用于图像显示装置的姿态欧拉角，并根据图像采集装置110采集的图像数据生成在该姿态下分别用于用户左眼和右眼的图像渲染数据。
在一些实施例中，图像处理装置120可以对接收的图像数据执行反畸变操作以抵消在图像显示装置130中经过光学元件成像时将产生的畸变，从而减少用户经过光学元件观看显示的图像时可能观察到的图像变形。
在一些实施例中，图像处理装置120可以对接收的图像或经过反畸变操作的图像执行多分辨率渲染操作，从而生成在图像显示装置130中要显示的图像。例如，图像处理装置120可以根据眼球追踪算法获取用户的注视点坐标值，并根据注视点坐标值获取用于对图像进行分割的目标像素的坐标值。然后，所述图像处理装置120可以根据上述目标像素的坐标值将图像分割成 多个子图像，并调整所述多个子图像中至少一个子图像的分辨率，使得图像中对应于用户注视点的子图像的分辨率高于其他子图像的分辨率。
在一些实施例中，图像处理装置120可以对上述目标像素的坐标值执行映射操作。所述目标像素的坐标值可以对应于第一数据范围，例如可以是(0,0)到(4320,4800)，即，所述坐标值可以具有处于第一数据范围内的数据值。所述图像像素值可以对应于第二数据范围，例如可以是(0,255)，即所述图像像素值可以具有处于第二数据范围内的数据值。所述映射操作，可以实现将对应于第一数据范围的数据值变换为对应于第二数据范围的数据值，例如可以通过基于第一数据范围和第二数据范围确定的变换参数来实现。
对目标像素的坐标值执行映射操作后获得的数据可以称为第一像素数据，所述第一像素数据对应于图像像素值的第二数据范围，即可以实现利用对应于第二数据范围的图像像素数据值来表示对应于第一数据范围的目标像素坐标数据值。
此外，由于所述经过映射操作获得的第一像素数据对应于图像像素值对应的第二数据范围，图像处理装置120可以将所述第一像素数据与要传送给图像显示装置130的用于显示的图像的第二像素数据一起合成为合成图像，并将所述合成图像传送给图像显示装置130用于解析并显示。例如，通过解析可以将合成图像中的第一像素数据与第二像素数据分离，即将表示目标像素坐标值的第一像素数据与要显示的图像的第二像素数据区分开。
这里的图像处理装置120可以实现为一个或多个专用或通用的计算机系统，例如，个人电脑、笔记本电脑、平板电脑、手机、个人数码助理(personal digital assistance，PDA)、智能眼镜、智能手表、智能指环、智能头盔及任何智能便携设备或可穿戴设备。图像处理装置120可以包括通信端口，与之相连的是实现数据通信的网络。图像处理装置120还可以包括至少一个处理器，用于执行程序指令。图像处理装置120可以包括一个内部通信总线。图像处理装置120可以包括不同形式的程序储存单元以及数据储存单元，例如硬盘、只读存储器(ROM)、随机存取存储器(RAM)，能够用于存储处理器处理和/或通信使用的各种数据文件，以及处理器所执行的可能的程序指令。图像处理装置120还可以包括一个输入/输出组件，支持图像处理装置120与其他组件(如用户界面)之间的输入/输出数据流。图像处理装置120也可以通过通信端口从网络发送和接收信息及数据。例如，图像处理装置120可以与图像 采集装置110、图像显示装置130以及姿态采集装置140进行相互的数据通信与传输。
图像显示装置130可以配置成接收来自图像处理装置120的图像数据。例如，图像显示装置130可以接收图像处理装置120生成的合成图像，并对其进行解析以获得其中用于对图像进行分割的目标像素的坐标值以及要显示的图像数据，例如，所述目标像素的坐标值可以通过对第一像素数据进行逆映射操作来实现。在一些实施例中，图像显示装置130可以包括驱动单元。驱动单元可以配置成对要显示的图像数据进行解析并获得用于对要显示的图像进行分割的目标像素的坐标值。基于解析得到的目标像素的坐标值，驱动单元可以对要显示的图像进行分割，对分割后的不同区域的图像分别进行拉伸并进行显示。
在一些实施例中，图像显示装置130可以是显示器、智能眼镜、智能头盔等。
尽管在图1A中，图像采集装置110、图像处理装置120、图像显示装置130以及姿态采集装置140呈现为单独的模块，本领域技术人员可以理解，上述装置模块可以被实现为单独的硬件设备，也可以被集成为一个或多个硬件设备。只要能够实现本公开描述的原理，不同的硬件设备的具体实现方式不应作为限制本公开保护范围的因素。
图1B示出了根据本公开的实施例的图像处理装置的示意性的框图。
在一些实施例中，图像处理装置120可以包括确定模块121，配置成确定用于对分别用于用户左眼和右眼的图像渲染数据进行图像分割的目标像素的坐标值。例如，当用户使用虚拟现实设备观看图像时，图像处理装置120可以根据眼球追踪算法得到用户的注视点坐标值。根据获取的用户注视点坐标值，图像处理装置120可以根据所述注视点坐标值确定所述图像中对应于注视点坐标值的注视点区域的边界。然后，图像处理装置120可以基于所述注视点区域的边界来确定用于对待处理的图像数据进行图像分割的目标像素的坐标值。例如，可以根据注视点坐标值确定作为高分辨率渲染的注视点区域。其中，注视点区域的边界点的坐标值可以作为用于对图像进行分割的目标像素的坐标值。
在一些实施例中，图像处理装置还可以包括变换模块122。例如，如前所述，图像处理装置120可以获取上述目标像素的坐标值，并利用变换模块122 对目标像素的坐标值执行映射操作，将目标像素的坐标值映射为第一像素数据。
对一个或多个目标像素的坐标值执行映射操作以获得第一像素数据指的是将目标像素的坐标值数据转换为使用图像像素值数据表示的数据。在根据本公开的一个实施例中，以4k图像为例，在4k图像中，目标像素的坐标值的数据范围在(0,0)到(4320,4800)之间，而图像像素值的数据一般是在(0,255)之间。在这里需要将坐标数据值转换到图像像素值对应的数据区间，所述转换可以通过变换参数来实现。例如，所述变换参数可以是(4800/255,255)之间的某个值，例如，变换参数可以是250。在目标像素的一个坐标值为4500的情况下，所述映射操作可以是利用坐标值4500除以变换参数250进行数据运算，得到的商为18，余数为0。在此处，所述商和余数可以称为第一像素数据，并且处于图像像素值的数据范围(0,255)之间。即，可以实现利用处于图像像素值的数据范围的第一像素数据来表示目标像素的坐标数据值。
在所述合成图像中，可以使用相邻的两个像素值中的第一位像素值表示坐标转换后的高位像素灰度值，第二位像素值表示坐标转换后的低位像素灰度值。其中，高位像素灰度值可以是目标像素的坐标值除以变换参数得到的商(例如，18)，低位像素灰度值可以是目标像素的坐标值除以变换参数得到的余数(例如，0)。在根据本公开的其他实施例中，还可以使用多个像素值来表示一个坐标值。
以此类推，目标像素的坐标值可以被映射为像素数据。当像素的坐标值是二维的形式时，其中每个维度的坐标值均可以采用如前所述的映射方法映射成像素数据。当像素的坐标值具有更多维度的参数时，也可以采用相同的方法将坐标值转换为像素数据。
如前所述，在将像素的坐标值转换为像素数据时需要确定变换参数。只要能够将坐标值的范围映射到图像像素值的范围内，这里的变换参数可以根据实际情况任意选择。例如，能够满足坐标值除以变换参数得到的商和余数的值处于图像像素值的范围即可。例如，当需要将(0,0)到(4320,4800)的坐标值范围映射到(0,255)的像素灰度值的范围内时，变换参数可以是(4800/255,255)之间的任意一个整数。例如，变换参数可以是250，或其他符合上述条件的可以使用像素灰度值表示的任何数。
进一步地，图像处理装置120还可以包括获取模块123，配置成获取要显 示的图像的像素数据作为第二像素数据。
进一步地，图像处理装置120还可以包括合成模块124，配置成合成第一像素数据与第二像素数据以获得合成图像。其中合成第一像素数据以及第二像素数据以获得合成图像的操作可以包括：新建一个图像作为合成图像，并将第一像素数据写入合成图像的第一行，并将第一行的剩余像素位用零补齐。之后，将第二像素数据写入合成图像。本领域技术人员可以理解，第一像素数据和第二像素数据的合成方式不限于以上示例。根据实际情况，可以使用任何组合方式合成第一像素数据以及第二像素数据。例如，可以先写入第二像素数据，随后再写入第一像素数据。又例如，可以在第一像素数据之前和/或第一像素数据和第二像素数据之间写入特定的用于标识的数据，使得图像显示装置可以识别并区分第一像素数据和第二像素数据。
图像处理装置120还可以包括传送模块125，其配置成将上述合成图像传送给图像显示装置130。
在一些实施例中，图像处理装置120还可以包括反畸变模块126。反畸变模块126可以根据图像显示装置的光学参数对上述分别用于用户左眼和右眼的图像渲染数据执行反畸变操作。即根据图像显示设备(如虚拟现实显示设备)的光学参数及屏幕坐标进行反畸变网格的绘制及纹理贴图，得到反畸变后的图像。当经过反畸变处理的图像通过对应的图像显示装置进行显示时，用户将可以观看到基本不产生畸变的图像。
在一些实施例中，图像处理装置120还可以包括多分辨率渲染模块127，其可以配置成根据目标像素将执行反畸变操作后的所述图像划分为多个子图像，并调整多个子图像中至少一个子图像的分辨率，使得图像的中心(或对应于用户注视点的区域)的分辨率高于其它子图像分辨率。
例如，当用户的注视点位于图像中心时可以确定图像的中心区域作为用户的注视点区域。其中，注视点区域可以确定为一个矩形区域。并且，利用矩形的注视点区域的四个顶点，可以将图像划分为九个区域。多分辨率渲染模块127可以对划分出的九个图像区域分别进行不同分辨率的渲染。例如，对应于注视点区域的图像区域可以具有最高的分辨率。对于其他图像区域可以根据与注视点区域的距离确定相应的分辨率。例如，当图像区域离注视点区域距离越远时，其分辨率可以设置成更低。
利用本公开的实施例提供的图像处理系统，可以实现坐标数据与图像数 据的同步传输，提高了图像传输过程中的效率，还可以实现对图像进行反畸变操作，减少用户观看图像时的图像变形，还可以对图像数据进行多分辨率渲染，减轻图像数据的传输压力。
图2示出了根据本公开的实施例的图像处理方法的示例性的流程图。如图2所示，根据图像处理方法200，在步骤S201中，确定图像中的一个或多个目标像素的坐标值，其中目标像素用于对图像进行分割。在步骤S202中，对一个或多个目标像素的坐标值执行映射操作以获得第一像素数据。在步骤S203中，读取图像以获得第二像素数据。在步骤S204中，合成第一像素数据以及第二像素数据以获得合成图像。在步骤S205中，将合成图像传送给显示装置的驱动单元，其中通过驱动单元读取合成图像中的第一像素数据，并对第一像素数据执行逆映射操作以获得一个或多个目标像素的坐标值，其中上述坐标值用于将所述图像分割成多个子图像并分别进行驱动显示。
在一些实施例中，在步骤S202中，对一个或多个目标像素的坐标值执行映射操作以获得第一像素数据指的是将目标像素的坐标值转换为可以使用图像的像素数据表示的数据。
图4示出了根据本公开实施例的映射操作的流程图，首先，在步骤S401，确定用于所述映射操作的变换参数，例如所述变换参数可以基于目标像素的数据范围和图像像素值的数据范围来确定。
接着，在步骤S402，对于所述一个或多个目标像素的坐标中的每一个的坐标值，基于所述变换参数将所述坐标值变换到所述图像的像素值范围之内以获得第一像素数据。其中，所述第一像素数据包括将所述坐标值除以所述变换参数得到的商和将所述坐标值除以所述变换参数得到的余数。
例如，以4k图像为例，在4k图像中像素的坐标值范围在(0,0)到(4320,4800)之间，而像素灰度值一般是在(0,255)之间，需要将坐标值转换到像素灰度值对应的区间。
在一些实施例中，可以使用相邻的两个像素值来表示第一像素数据，即利用相邻的两个像素值来表示目标像素的一个坐标值。所述相邻的两个像素值中的第一位像素值表示坐标转换后的高位像素灰度值，第二位像素值表示坐标转换后的低位像素灰度值。其中，高位像素灰度值可以是目标像素的坐标值除以变换参数得到的商，低位像素灰度值可以是目标像素的坐标值除以变换参数得到的余数。根据本公开的其他实施例，还可以使用多个像素值来 表示目标像素的一个坐标值。
以此类推，目标像素的坐标值可以被映射为像素数据。当像素的坐标值是二维的形式时，其中每个维度的坐标值均可以采用如前所述的映射方法映射成像素数据。当像素的坐标值具有更多维度的参数时，也可以采用相同的方法将坐标值转换为像素数据。
如前所述，在将像素的坐标值转换为像素数据时需要确定变换参数。只要能够将坐标值的范围映射到像素灰度值的范围内，这里的变换参数可以根据实际情况任意选择，能够满足坐标值除以变换参数得到的商和余数的值落入像素灰度值的范围即可。例如，当需要将(0,0)到(4320,4800)的坐标值范围映射到(0,255)的范围内时，变换参数可以是(4800/255,255)之间的任意一个整数。
在一些实施例中，在步骤S204中，合成第一像素数据以及第二像素数据以获得合成图像包括：建立一个新图像；将第一像素数据写入新图像的第一行；在第一像素数据之后写入第二像素数据；将写入数据后的新图像作为合成图像。例如，将第一像素数据写入合成图像的第一行，并将第一行的剩余像素位用零补齐。之后，将第二像素数据写入合成图像。本领域技术人员可以理解，第一像素数据和第二像素数据的合成方式不限于以上示例。根据实际情况，可以使用任何组合方式合成第一像素数据以及第二像素数据。例如，可以先写入第二像素数据，随后再写入第一像素数据。又例如，可以在第一像素数据之前和/或第一像素数据和第二像素数据之间写入特定的用于标识的数据，使得图像显示装置可以识别并区分第一像素数据和第二像素数据。
在一些实施例中，图像处理方法200还可以包括在确定一个或多个目标像素的坐标值之前，确定将用于显示图像的显示装置的光学参数；基于显示装置的光学参数对图像执行反畸变操作。
其中，上述反畸变操作可以包括：根据图像显示装置的光学参数确定反畸变网格，以及根据图像的坐标将图像的纹理贴在所述反畸变网格上。
例如，反畸变操作可以包括根据图像采集设备采集的图像数据建立顶点数组、三角形数组以及图像UV坐标数组。根据图像显示设备的光学参数，可以对原始图像数据的坐标进行坐标变换，并根据坐标变换绘制根据反畸变操作所生成的网格。确定反畸变操作生成的网格后，可以将图像纹理贴在上述生成的网格上，从而生成经过反畸变操作后的图像。
回到图2，在一些实施例中，图像处理方法200还可以包括根据目标像素将执行反畸变操作后的图像划分为多个子图像，并调整多个子图像中至少一个子图像的分辨率，例如，使得图像的中心分辨率高于四周分辨率。下文中将参考图3C描述多分辨率渲染的过程。
图3A-3F示出了利用本公开的实施例的图像处理系统的执行的图像处理流程的示意图。
图3A示出的是通过图像采集装置获取的原始图像数据的示意图。其中原始图像数据可以是图像采集装置直接采集的图像数据，也可以是图像处理装置利用前述的姿态数据获取的图像数据。
图3B示出的是经过反畸变处理后的图像数据的示意图。其中，反畸变操作是根据相应的显示设备的光学参数确定的。也就是说，针对不同的显示设备，反畸变操作的参数可以是不同的。经过反畸变操作后，图像数据被转换为如图3B示出的存在畸变的图像。
图3C示出的是经过多分辨率渲染后的图像数据的示意图。经过反畸变操作后，系统100还可以根据眼球追踪算法获得用户的注视点坐标值，并根据注视点坐标值对进行反畸变操作后的图像进行多分辨率渲染操作。
根据本公开实施例的图像处理方法，可以根据确定的注视点坐标值来确定图像中的一个或多个目标像素的坐标值，所述目标像素用于对进行反畸变操作后的图像进行分割，以用于进行多分辨率渲染操作。
图5示出了根据本公开实施例的确定目标像素的坐标值的流程图。首先，在步骤S501，根据眼球追踪算法获取用户的注视点坐标值，例如可以根据眼球追踪算法获取得到用户的注视点位于图像中心，所述图像中心的坐标值则可以作为用户的注视点坐标值。
接着，在步骤S502，根据所述注视点坐标值来确定所述图像中对应于注视点坐标值的注视点区域的边界。所述注视点区域可以为以注视点为中心的矩形区域。所述矩形区域的边界即为注视点区域的边界。
然后，在步骤S503，基于所述注视点区域的边界确定目标像素的坐标值。例如，可以将矩形的注视点区域的四个顶点确定为所述目标像素。
例如，当用户的注视点位于图像中心时可以确定图像的中心区域作为用户的注视区域。如图3C中所示出的，注视区域可以确定为一个矩形区域。并且，利用矩形的注视区域的四个顶点，可以将图像划分为九个区域。在图3C 中，位于左侧的图像对应于左眼图像数据，位于右侧的图像对应于右眼图像数据。例如，所述左眼图像数据可以被分割成1-9号图像区域，以及，所述右眼图像数据可以被分割成1’-9’号图像区域。系统100可以对图3C中示出的分割得到的图像区域分别地进行不同分辨率的渲染。
例如，以图3C中的左眼图像数据为例，位于中心的5号图像区域可以具有最高的分辨率。位于5号图像区域的左右两侧的4号图像区域和6号图像区域在竖直方向上(即1号图像区域与7号图像区域的连线方向)的分辨率可以与5号图像区域的分辨率相同，而在水平方向(即4号图像区域与6号图像区域的连线方向)的分辨率可以小于5号图像区域的分辨率。例如，可以是5号图像区域的分辨率的一半(或1/3，或根据情况选取任何分辨率，在本公开中以分辨率下降为一半作为示例)。类似地，位于5号图像区域的上下两侧的2号图像区域和8号图像区域在水平方向上的分辨率可以与5号图像区域的分辨率相同，在竖直方向上的分辨率可以小于5号图像区域的分辨率，例如，可以是5号图像区域内分辨率的一半。对于位于5号图像区域的左上方、右上方、左下方以及右下方的1号、3号、7号与9号图像区域，上述图像区域在水平方向上和竖直方向上的图像分辨率可以均小于5号图像区域的分辨率。例如，1号、3号、7号与9号图像区域在水平方向上和竖直方向上的图像分辨率可以均是5号图像区域内分辨率的一半。
经过图3B示出的反畸变操作和图3C示出的多分辨率渲染操作，系统100生成用于显示的图像数据。
图3D示出的是包括目标像素坐标数据与要显示的图像数据的合成图像的示意图。在一些实施例中，合成图像比要进行显示的图像(如图3C中的图像)多一行像素数据。例如，如果图3C中的图像数据包括m行*n列个像素，那么合成图像可以包括(m+1)行*n列个像素的图像数据。如图3D中所示出的，合成图像中多出的一行像素用于存储所述目标像素的坐标值。例如，合成图像的第一行像素的数据包括通过对目标像素的坐标值执行映射操作获得的第一像素数据301和第一像素数据302。其中，图3D中左侧图像中的第一像素数据301可以对应于通过对图3C中左侧图像中的目标像素的坐标值执行映射操作获得的；图3D中右侧图像中的第一像素数据302可以对应于通过对图3C中右侧图像中的目标像素的坐标值执行映射操作获得的。在合成图像中，可以从第二行像素开始写入系统100生成的用于显示的图像数据(例如， 如图3C所示的图像数据)。
图3E示出的是图像显示装置中显示的图像的示意图。其中，图像显示装置可以读取合成图像，并从中解析出目标像素的坐标值以及用于显示的图像数据。根据目标像素的坐标值，图像显示装置将确定要显示的图像中的多个子区域，并针对每个子区域中的不同分辨率的图像分别进行拉伸与显示。
图3F示出的是用户通过显示设备观看到的图像的示意图。由于显示设备中采用的是光学元件成像，因此要显示的图像数据在经过光学元件后会产生一定的畸变。由于如图3B中所示出的，原始数据图像已经经过了反畸变处理，因此当用户通过显示设备观看图像时，显示设备的光学元件产生的畸变与反畸变处理的效果相抵消，用户可以观看到没有畸变或畸变很小的图像。
利用本公开的实施例提供的图像处理方法，可以实现目标像素的坐标数据与要显示的图像数据的同步传输，提高了图像传输的效率，还可以实现对图像进行反畸变操作，减少用户观看图像时的图像变形，还可以对图像数据进行多分辨率渲染，减轻图像数据的传输压力。
根据本公开的另一方面，还提供了一种计算机可读存储介质，其上存储有计算机可读的指令，当利用计算机执行所述指令时可以执行上述图像处理方法。
利用本公开提供的图像处理方法、图像处理装置、图像处理系统，可以实现坐标数据与图像数据的同步传输，提高了图像传输过程中的效率。此外，可以利用眼球追踪算法准确定位观察区域，并只着重渲染该区域，就可以节省较大的工作量。由于硬件接收数据量有限，多分辨率渲染可以减少PC传递给显示屏的数据传输量，减轻数据传输压力。
为了实现不同的模块、单元以及它们在本公开中所描述的功能，计算机硬件平台可以被用作以上描述的一个或多个元素的硬件平台。这类计算机的硬件元素、操作系统和程序语言是常见的，可以假定本领域技术人员对这些技术都足够熟悉，能够利用这里描述的技术提供运动控制所需要的信息。一台包含用户界面(user interface，UI)元素的计算机能够被用作个人计算机(personal computer，PC)或其他类型的工作站或终端设备，被适当程序化后也可以作为服务器使用。可以认为本领域技术人员对这样的结构、程序以及这类计算机设备的一般操作都是熟悉的，因此所有附图也都不需要额外的解释。
以上概述了提供运动控制所需要的信息的方法的不同方面和/或通过程序实现其他步骤的方法。技术中的程序部分可以被认为是以可执行的代码和/或相关数据的形式而存在的“产品”或“制品”，通过计算机可读的介质所参与或实现的。有形的、永久的储存介质可以包括任何计算机、处理器、或类似设备或相关的模块所用到的内存或存储器。例如，各种半导体存储器、磁带驱动器、磁盘驱动器或者类似任何能够为软件提供存储功能的设备。
所有软件或其中的一部分有时可能会通过网络进行通信，如互联网或其他通信网络。此类通信可以将软件从一个计算机设备或处理器加载到另一个。例如：从运动控制系统的一个服务器或主机计算机加载至一个计算机环境的硬件平台，或其他实现系统的计算机环境，或与提供运动控制所需要的信息相关的类似功能的系统。因此，另一种能够传递软件元素的介质也可以被用作局部设备之间的物理连接，例如光波、电波、电磁波等，通过电缆、光缆或者空气等实现传播。用来载波的物理介质如电缆、无线连接或光缆等类似设备，也可以被认为是承载软件的介质。在这里的用法除非限制了有形的“储存”介质，其他表示计算机或机器“可读介质”的术语都表示在处理器执行任何指令的过程中参与的介质。
同时，本公开使用了特定词语来描述本公开的实施例。如“一个实施例”、“一实施例”、和/或“一些实施例”意指与本公开至少一个实施例相关的某一特征、结构或特点。因此，应强调并注意的是，本说明书中在不同位置两次或多次提及的“一实施例”或“一个实施例”或“一替代性实施例”并不一定是指同一实施例。此外，本公开的一个或多个实施例中的某些特征、结构或特点可以进行适当的组合。
此外，本领域技术人员可以理解，本公开的各方面可以通过若干具有可专利性的种类或情况进行说明和描述，包括任何新的和有用的工序、机器、产品或物质的组合，或对他们的任何新的和有用的改进。相应地，本公开的各个方面可以完全由硬件执行、可以完全由软件(包括固件、常驻软件、微码等)执行、也可以由硬件和软件组合执行。以上硬件或软件均可被称为“数据块”、“模块”、“引擎”、“单元”、“组件”或“系统”。此外，本公开的各方面可能表现为位于一个或多个计算机可读介质中的计算机产品，该产品包括计算机可读程序编码。
计算机可读信号介质可能包含一个内含有计算机程序编码的传播数据信 号，例如在基带上或作为载波的一部分。该传播信号可能有多种表现形式，包括电磁形式、光形式等等、或合适的组合形式。计算机可读信号介质可以是除计算机可读存储介质之外的任何计算机可读介质，该介质可以通过连接至一个指令执行系统、装置或设备以实现通讯、传播或传输供使用的程序。位于计算机可读信号介质上的程序编码可以通过任何合适的介质进行传播，包括无线电、电缆、光纤电缆、射频信号、或类似介质、或任何上述介质的组合。
除非另有定义，这里使用的所有术语(包括技术和科学术语)具有与本公开所属领域的普通技术人员共同理解的相同含义。还应当理解，诸如在通常字典里定义的那些术语应当被解释为具有与它们在相关技术的上下文中的含义相一致的含义，而不应用理想化或极度形式化的意义来解释，除非这里明确地这样定义。
上面是对本公开的说明，而不应被认为是对其的限制。尽管描述了本公开的若干示例性实施例，但本领域技术人员将容易地理解，在不背离本公开的新颖教学和优点的前提下可以对示例性实施例进行许多修改。因此，所有这些修改都意图包含在权利要求书所限定的本公开范围内。应当理解，上面是对本公开的说明，而不应被认为是限于所公开的特定实施例，并且对所公开的实施例以及其他实施例的修改意图包含在所附权利要求书的范围内。本公开由权利要求书及其等效物限定。

Claims (20)

  1. 一种图像处理方法，包括：
    确定图像中的一个或多个目标像素的坐标值，其中所述目标像素用于对所述图像进行分割；
    对所述一个或多个目标像素的坐标值执行映射操作以获得第一像素数据；
    获取所述图像的像素数据作为第二像素数据；
    合成所述第一像素数据以及所述第二像素数据以获得合成图像。
  2. 如权利要求1所述的图像处理方法，其中，确定图像中的一个或多个目标像素的坐标值包括：
    根据眼球追踪算法获取用户的注视点坐标值；
    根据所述注视点坐标值确定所述图像中对应于注视点坐标值的注视点区域的边界；以及
    基于所述注视点区域的边界确定目标像素的坐标值。
  3. 如权利要求1或2所述的图像处理方法，其中，所述对所述一个或多个目标像素的坐标值执行映射操作以获得第一像素数据包括：
    确定用于所述映射操作的变换参数；
    对于所述一个或多个目标像素中的每一个的坐标值，基于所述变换参数将所述坐标值变换到所述图像的像素值范围之内，以获得第一像素数据，
    其中，所述第一像素数据包括将所述坐标值除以所述变换参数得到的商和将所述坐标值除以所述变换参数得到的余数。
  4. 如权利要求3所述的图像处理方法，还包括，使用相邻的两个像素值表示所述第一像素数据，
    其中所述相邻的两个像素值包括第一位像素值和第二位像素值，其中：
    将所述坐标值除以所述变换参数得到的商作为所述相邻的两个像素值中的第一位像素值；
    将所述坐标值除以所述变换参数得到的余数作为所述相邻的两个像素值中的第二位像素值。
  5. 如权利要求1-4中任一项所述的图像处理方法，其中，合成所述第一像素数据以及所述第二像素数据以获得合成图像包括：
    建立一个新图像；
    将所述第一像素数据写入所述新图像；
    在所述第一像素数据之后写入所述第二像素数据；
    将写入数据后的所述新图像作为合成图像。
  6. 如权利要求1-5中任一项所述的图像处理方法，还包括：
    在确定所述一个或多个目标像素的坐标值之前，确定将用于显示所述图像的显示装置的光学参数；
    基于所述显示装置的光学参数对所述图像执行反畸变操作。
  7. 如权利要求6所述的图像处理方法，其中，所述反畸变操作包括：
    根据所述显示装置的光学参数确定反畸变网格；
    基于所述反畸变网格生成经过反畸变操作后的图像。
  8. 如权利要求1-7中任一项所述的图像处理方法，还包括：
    基于所述图像中的一个或多个目标像素的坐标值，将所述图像分割成多个子图像，并调整所述多个子图像中的至少一个子图像的分辨率，使得所述多个子图像中至少一个子图像的分辨率高于其他子图像的分辨率。
  9. 如权利要求1所述的图像处理方法，
    还包括：
    将所述合成图像传送给显示装置的驱动单元，其中通过所述驱动单元读取所述合成图像中的第一像素数据，并对所述第一像素数据执行逆映射操作以获得所述一个或多个目标像素的坐标值，所述坐标值用于将所述图像分割成多个子图像并分别进行驱动显示。
  10. 一种图像处理装置，包括：
    确定模块，配置成确定图像中的一个或多个目标像素的坐标值，其中所述目标像素用于对所述图像进行分割；
    变换模块，配置成对所述确定模块确定的所述一个或多个目标像素的坐标值执行映射操作以获得第一像素数据；
    获取模块，配置成获取所述图像的像素数据作为第二像素数据；以及
    合成模块，配置成合成所述变换模块获得的所述第一像素数据以及所述获取模块获取的所述第二像素数据以获得合成图像。
  11. 根据权利要求10所述的图像处理装置，其中，所述确定模块根据眼球追踪算法获取用户的注视点坐标值；根据所述注视点坐标值确定所述图像 中对应于注视点坐标值的注视点区域的边界；以及基于所述注视点区域的边界确定目标像素的坐标值。
  12. 根据权利要求10或11所述的图像处理装置，其中，所述变换模块确定用于所述映射操作的变换参数；对于所述一个或多个目标像素中的每一个的坐标值，基于所述变换参数将所述坐标值变换到所述图像的像素值范围之内，以获得第一像素数据，其中，
    所述第一像素数据包括将所述坐标值除以所述变换参数得到的商和将所述坐标值除以所述变换参数得到的余数。
  13. 根据权利要求10-12中任一项所述的图像处理装置，其中，所述变换模块还配置成使用相邻的两个像素值表示所述第一像素数据，
    其中所述相邻的两个像素值包括第一位像素值和第二位像素值，其中：
    将所述坐标值除以所述变换参数得到的商作为所述相邻的两个像素值中的第一位像素值；
    将所述坐标值除以所述变换参数得到的余数作为所述相邻的两个像素值中的第二位像素值。
  14. 根据权利要求10-13中任一项所述的图像处理装置，其中，所述合成模块建立一个新图像；将所述第一像素数据写入所述新图像；在所述第一像素数据之后写入所述第二像素数据；将写入数据后的所述新图像作为合成图像。
  15. 根据权利要求10-14中任一项所述的图像处理装置，还包括：
    反畸变模块，配置成在确定所述一个或多个目标像素的坐标值之前，确定将用于显示所述图像的显示装置的光学参数，并基于所述显示装置的光学参数对所述图像执行反畸变操作。
  16. 根据权利要求10-15中任一项所述的图像处理装置，其中，所述反畸变模块配置成根据所述显示装置的光学参数确定反畸变网格；基于所述反畸变网格生成经过反畸变操作后的图像。
  17. 根据权利要求10-16中任一项所述的图像处理装置，还包括：
    多分辨率渲染模块，配置成根据所述目标像素将所述图像划分为多个子图像，并调整所述多个子图像中的至少一个子图像的分辨率，使得所述多个子图像中至少一个子图像的分辨率高于其他子图像的分辨率。
  18. 如权利要求10-17中任一项所述的图像处理装置，还包括：
    传送模块，将所述合成图像传送给显示装置的驱动单元，其中通过所述驱动单元读取所述合成图像中的第一像素数据，并对所述第一像素数据执行逆映射操作以获得所述一个或多个目标像素的坐标值，所述坐标值用于将所述图像分割成多个子图像并分别进行驱动显示。
  19. 一种计算机可读存储介质，其上存储有计算机可读的指令，当利用计算机执行所述指令时执行如权利要求1-9中任一项所述的图像处理方法。
  20. 一种图像处理系统，包括：
    图像采集装置，配置成采集图像；
    图像显示装置，包括驱动单元；以及
    图像处理装置，包括，
    接收模块，配置成从所述图像采集装置接收所述图像；
    确定模块，配置成确定所述图像中的一个或多个目标像素的坐标值，其中所述目标像素用于对所述图像进行分割；
    变换模块，配置成对所述一个或多个目标像素的坐标值执行映射操作以获得第一像素数据；
    获取模块，配置成获取所述图像的像素数据作为第二像素数据；
    合成模块，配置成合成所述第一像素数据以及所述第二像素数据以获得合成图像；以及
    传送模块，配置成将所述合成图像传送给所述图像显示装置，
    其中通过所述驱动单元读取所述合成图像中的第一像素数据，并对所述第一像素数据执行逆映射操作以获得所述一个或多个目标像素的坐标值，所述坐标值用于将所述图像分割成多个子图像并分别进行驱动显示。
PCT/CN2019/078014 2018-03-19 2019-03-13 图像处理方法、图像处理装置、图像处理系统及介质 WO2019179342A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/500,538 US11127126B2 (en) 2018-03-19 2019-03-13 Image processing method, image processing device, image processing system and medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810225841.0A CN110290285B (zh) 2018-03-19 2018-03-19 图像处理方法、图像处理装置、图像处理系统及介质
CN201810225841.0 2018-03-19

Publications (1)

Publication Number Publication Date
WO2019179342A1 true WO2019179342A1 (zh) 2019-09-26

Family

ID=67986716

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/078014 WO2019179342A1 (zh) 2018-03-19 2019-03-13 图像处理方法、图像处理装置、图像处理系统及介质

Country Status (3)

Country Link
US (1) US11127126B2 (zh)
CN (1) CN110290285B (zh)
WO (1) WO2019179342A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111447339A (zh) * 2020-03-26 2020-07-24 西安万像电子科技有限公司 图像传输方法和系统

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11544903B2 (en) * 2019-12-13 2023-01-03 Sony Group Corporation Reducing volumetric data while retaining visual fidelity
CN115150390B (zh) * 2022-06-27 2024-04-09 山东信通电子股份有限公司 一种图像显示方法、装置、设备及介质
CN116469091A (zh) * 2023-04-21 2023-07-21 浪潮智慧科技有限公司 一种基于实时视频的水尺自动读数方法、设备及介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103747236A (zh) * 2013-12-30 2014-04-23 中航华东光电有限公司 结合人眼跟踪的立体视频处理系统及方法
US20140267602A1 (en) * 2013-03-14 2014-09-18 CSR Technology, Inc. System and method for real time 2d to 3d conversion of a video in a digital camera
CN105812777A (zh) * 2015-01-21 2016-07-27 成都理想境界科技有限公司 双目ar头戴显示设备及其信息显示方法
CN107622474A (zh) * 2017-09-26 2018-01-23 北京大学深圳研究生院 基于主视点的全景视频映射方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7496242B2 (en) * 2004-12-16 2009-02-24 Agfa Inc. System and method for image transformation
CN100399835C (zh) * 2005-09-29 2008-07-02 北京理工大学 用于现场数字三维重建的增强现实定点观察系统
CN102708585B (zh) * 2012-05-09 2015-05-20 北京像素软件科技股份有限公司 一种渲染模型轮廓边缘的方法
KR101921672B1 (ko) * 2014-10-31 2019-02-13 후아웨이 테크놀러지 컴퍼니 리미티드 이미지 처리 방법 및 장치
IN2015CH02866A (zh) * 2015-06-09 2015-07-17 Wipro Ltd
KR102558473B1 (ko) * 2016-09-30 2023-07-25 삼성전자주식회사 이미지를 표시하기 위한 방법 및 그 전자 장치
WO2018129234A1 (en) * 2017-01-05 2018-07-12 Lang Philipp K Improved accuracy of displayed virtual data with optical head mount displays for mixed reality
US10638030B1 (en) * 2017-01-31 2020-04-28 Southern Methodist University Angular focus stacking
US20180314066A1 (en) * 2017-04-28 2018-11-01 Microsoft Technology Licensing, Llc Generating dimming masks to enhance contrast between computer-generated images and a real-world view

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140267602A1 (en) * 2013-03-14 2014-09-18 CSR Technology, Inc. System and method for real time 2d to 3d conversion of a video in a digital camera
CN103747236A (zh) * 2013-12-30 2014-04-23 中航华东光电有限公司 结合人眼跟踪的立体视频处理系统及方法
CN105812777A (zh) * 2015-01-21 2016-07-27 成都理想境界科技有限公司 双目ar头戴显示设备及其信息显示方法
CN107622474A (zh) * 2017-09-26 2018-01-23 北京大学深圳研究生院 基于主视点的全景视频映射方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111447339A (zh) * 2020-03-26 2020-07-24 西安万像电子科技有限公司 图像传输方法和系统

Also Published As

Publication number Publication date
US11127126B2 (en) 2021-09-21
CN110290285A (zh) 2019-09-27
CN110290285B (zh) 2021-01-22
US20210118109A1 (en) 2021-04-22

Similar Documents

Publication Publication Date Title
KR102458339B1 (ko) 360도 3d 입체 영상을 생성하는 전자 장치 및 이의 방법
US20200051269A1 (en) Hybrid depth sensing pipeline
WO2019179342A1 (zh) 图像处理方法、图像处理装置、图像处理系统及介质
US10692274B2 (en) Image processing apparatus and method
CN107925755B (zh) 针对图像处理进行平面表面检测的方法和系统
CN107274338B (zh) 用于深度图的低等待时间变形的系统、方法和装置
JP2020507850A (ja) 画像内の物体の姿の確定方法、装置、設備及び記憶媒体
WO2019238114A1 (zh) 动态模型三维重建方法、装置、设备和存储介质
CN109683699B (zh) 基于深度学习实现增强现实的方法、装置及移动终端
US10148875B1 (en) Method and system for interfacing multiple channels of panoramic videos with a high-definition port of a processor
US11044398B2 (en) Panoramic light field capture, processing, and display
CN108430032B (zh) 一种实现vr/ar设备位置共享的方法及设备
US11308588B2 (en) Image processing method, device, and virtual reality display apparatus
CN113628276A (zh) 经由深度学习的单图像超广角鱼眼相机校准
CN112055869A (zh) 对面部的透视畸变校正
WO2023207379A1 (zh) 图像处理方法、装置、设备及存储介质
US11985294B2 (en) Information processing apparatus, information processing method, and program
US20150077575A1 (en) Virtual camera module for hybrid depth vision controls
CN115908120B (zh) 图像处理方法和电子设备
US11636708B2 (en) Face detection in spherical images
KR20200103278A (ko) 뷰 방향이 표시되는 vr 컨텐츠 제공 시스템 및 방법
US20210303824A1 (en) Face detection in spherical images using overcapture
CN114257733A (zh) 具有视点偏移的全向图像的图像处理的方法和系统
KR102534449B1 (ko) 이미지 처리 방법, 장치, 전자 장치 및 컴퓨터 판독 가능 저장 매체
CN116453131B (zh) 文档图像矫正方法、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19771061

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 29.01.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19771061

Country of ref document: EP

Kind code of ref document: A1