CN115525152A

CN115525152A - Image processing method, system, device, electronic equipment and storage medium

Info

Publication number: CN115525152A
Application number: CN202211139797.4A
Authority: CN
Inventors: 向阳; 张波; 吕涛
Original assignee: Zejing Xi'an Automotive Electronics Co ltd
Current assignee: Zejing Xi'an Automotive Electronics Co ltd
Priority date: 2022-09-19
Filing date: 2022-09-19
Publication date: 2022-12-27

Abstract

The disclosure provides an image processing method, an image processing system, an image processing device, electronic equipment and a computer-readable storage medium, and relates to the technical field of augmented reality. The method comprises the following steps: determining the current fixation point coordinate, wherein the current fixation point coordinate is used for indicating the coordinate of the current fixation position of human eyes; performing coordinate conversion on the current fixation point coordinate in response to the position change instruction corresponding to the detected current fixation point coordinate to obtain a target fixation point coordinate of human eyes relative to target equipment; acquiring an augmented reality image, and determining image adjustment parameters of the augmented reality image according to the coordinates of the target fixation point; and carrying out position adjustment processing on the augmented reality image based on the image adjustment parameters so as to enable the augmented reality image to be in the visible range of human eyes. The display position of the augmented reality image can be automatically adjusted according to the position change of human eyes, so that the augmented reality image is always in the visual range of the human eyes, and manual operation is not needed.

Description

Image processing method, system, device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of augmented reality technologies, and in particular, to an image processing method, an image processing system, an image processing apparatus, an electronic device, and a computer-readable storage medium.

Background

Augmented Reality Head-Up Display (AR-HUD) is a vehicle-mounted interactive system integrating an Augmented Reality technology, a Head-Up Display technology and a multi-information fusion technology. The AR-HUD still uses the windshield as a display screen compared to a conventional HUD, but can provide a more remote virtual image and perform image rendering in combination with a real scene.

The continuous development of the HUD is not only due to driving safety and display convenience, but also due to the expansion of the AR technology, so that the HUD using scene has more possibilities. However, due to the small angle of view of the AR-HUD, the change in the sitting posture of the driver, and the like, the problem of insufficient display of the virtual image plane is likely to occur, and at this time, the user is required to manually perform operation adjustment; in addition, the AR-HUD can also cause the problem of non-corresponding virtual and real images, so that the deviation between the virtual image and the real scene target is caused, and unsafe factors are brought to the driving process.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The present disclosure is directed to an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium, so as to overcome, at least to some extent, the problems that the operation of the existing head-up display implementation scheme is time-consuming and labor-consuming, and it is difficult for a user to adjust a display image to a suitable visual range.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the invention.

According to a first aspect of the present disclosure, there is provided an image processing method including: determining the coordinates of a current fixation point, wherein the coordinates of the current fixation point are used for indicating the coordinates of the current fixation position of human eyes; performing coordinate conversion on the current fixation point coordinate in response to the detection of the position change instruction corresponding to the current fixation point coordinate to obtain a target fixation point coordinate of the human eye relative to target equipment; acquiring an augmented reality image, and determining image adjustment parameters of the augmented reality image according to the target fixation point coordinates; and carrying out position adjustment processing on the augmented reality image based on the image adjustment parameters so as to enable the augmented reality image to be in the visible range of the human eyes.

In an exemplary embodiment of the present disclosure, the determining the current gaze point coordinate comprises: acquiring the face image through an image acquisition device; acquiring a pre-trained facial feature extraction model, wherein the facial feature extraction model is obtained by training a facial image training set; and performing feature extraction on the face image through the face feature extraction model to obtain human eye information corresponding to the face image, wherein the human eye information comprises the current fixation point coordinate.

In an exemplary embodiment of the present disclosure, reference gaze point coordinates corresponding to a pre-stored reference position of a human eye are obtained; comparing the current fixation point coordinate with the reference fixation point coordinate to obtain a coordinate change difference value corresponding to the current fixation point coordinate; and if the coordinate change difference value is greater than or equal to a pre-configured coordinate change threshold value, generating the position change instruction.

In an exemplary embodiment of the present disclosure, the performing coordinate transformation on the current gaze point coordinate to obtain a target gaze point coordinate of the human eye relative to a target device includes: performing physical coordinate conversion on the fixation point pixel coordinates to obtain fixation point physical coordinates corresponding to the fixation point pixel coordinates; performing three-dimensional coordinate conversion on the physical coordinates of the fixation point to obtain initial three-dimensional fixation point coordinates corresponding to the physical coordinates of the fixation point; and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate to obtain the target fixation point coordinate.

In an exemplary embodiment of the present disclosure, the gaze point pixel coordinates comprise left eye pixel coordinates and right eye pixel coordinates; the physical coordinate conversion of the fixation point pixel coordinate to obtain the fixation point physical coordinate corresponding to the fixation point pixel coordinate comprises the following steps: determining a central point pixel coordinate corresponding to the human eyes according to the left eye pixel coordinate and the right eye pixel coordinate; and performing the physical coordinate conversion on the pixel coordinate of the central point to obtain the physical coordinate of the fixation point.

In an exemplary embodiment of the present disclosure, the performing three-dimensional coordinate conversion on the physical coordinates of the gaze point to obtain initial three-dimensional coordinates of the gaze point corresponding to the physical coordinates of the gaze point includes: acquiring an image equipment coordinate system corresponding to image acquisition equipment; determining a first transformation matrix matching the image device coordinate system; and performing three-dimensional coordinate conversion on the physical coordinates of the fixation point based on the first conversion matrix to obtain the initial three-dimensional fixation point coordinates.

In an exemplary embodiment of the present disclosure, the target device includes a driving device; the performing device mapping coordinate conversion on the initial three-dimensional fixation point coordinate to obtain the target fixation point coordinate comprises: acquiring a driving equipment coordinate system corresponding to the driving equipment; determining a second conversion matrix corresponding to the coordinate system of the driving equipment, wherein the second conversion matrix is determined based on the physical size information of the driving equipment and the equipment calibration information of the image acquisition equipment; and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate based on the second conversion matrix to obtain the target fixation point coordinate.

In an exemplary embodiment of the present disclosure, the image adjustment parameters include a translation matrix and a rotation matrix, and the performing the position adjustment processing on the augmented reality image based on the image adjustment parameters includes: acquiring image display parameters of the augmented reality image, wherein the image display parameters comprise a current display coordinate and a current rotation angle; and carrying out translation processing on the current display coordinates of the augmented reality image based on the translation matrix, and adjusting the current rotation angle of the augmented reality image based on the rotation matrix so as to adjust the display position of the augmented reality image.

In an exemplary embodiment of the present disclosure, the image adjustment parameters include a translation matrix and a rotation matrix, and the performing the position adjustment processing on the augmented reality image based on the image adjustment parameters includes: acquiring reflector parameters of an image reflector, wherein the image reflector is used for reflecting and displaying the augmented reality image; adjusting the reflector parameters according to the translation matrix and the rotation matrix so as to adjust the position of the image reflector; and displaying the augmented reality image based on the image reflector after the position adjustment processing so as to adjust the display position of the augmented reality image.

According to a second aspect of the present disclosure, there is provided an image processing system comprising: the image acquisition equipment is used for acquiring a face image and determining the current fixation point coordinate corresponding to the human eyes based on the face image; the coordinate conversion module is used for carrying out coordinate conversion processing on the current fixation point coordinate to obtain a target fixation point coordinate of the human eye relative to the target equipment; the transmission module is used for transmitting the target fixation point coordinates to the image position adjusting module; the image position adjusting module is used for determining image adjusting parameters of the augmented reality image based on the target fixation point coordinates; and performing position adjustment processing on the augmented reality image based on the image adjustment parameters so that the augmented reality image is positioned in the visible range of the human eyes.

According to a third aspect of the present disclosure, there is provided an image processing apparatus comprising: the current coordinate determination module is used for determining the coordinates of a current fixation point, and the coordinates of the current fixation point are used for indicating the coordinates of the current fixation position of human eyes; the target coordinate determination module is used for responding to the position change instruction corresponding to the detected current fixation point coordinate, and performing coordinate conversion on the current fixation point coordinate to obtain a target fixation point coordinate of the human eye relative to the target equipment; the adjustment parameter determining module is used for acquiring an augmented reality image and determining image adjustment parameters of the augmented reality image according to the target fixation point coordinates; and the image adjusting module is used for carrying out position adjustment processing on the augmented reality image based on the image adjusting parameters so as to enable the augmented reality image to be in the visible range of the human eyes.

In an exemplary embodiment of the present disclosure, the target coordinate determination module includes a target coordinate determination unit to: acquiring the face image through image acquisition equipment; acquiring a pre-trained facial feature extraction model, wherein the facial feature extraction model is obtained by training a facial image training set; and extracting the features of the face image through the face feature extraction model to obtain the eye information corresponding to the face image, wherein the eye information comprises the current fixation point coordinate.

In an exemplary embodiment of the present disclosure, the image processing apparatus further includes a coordinate change detection module, configured to acquire a reference gaze point coordinate corresponding to a pre-stored reference position of the human eye; comparing the current fixation point coordinate with the reference fixation point coordinate to obtain a coordinate change difference value corresponding to the current fixation point coordinate; and if the coordinate change difference value is greater than or equal to a pre-configured coordinate change threshold value, generating the position change instruction.

In an exemplary embodiment of the present disclosure, the current gaze point coordinates comprise gaze point pixel coordinates, the target coordinate determination module comprises a target coordinate determination unit to: performing physical coordinate conversion on the fixation point pixel coordinates to obtain fixation point physical coordinates corresponding to the fixation point pixel coordinates; performing three-dimensional coordinate conversion on the fixation point physical coordinate to obtain an initial three-dimensional fixation point coordinate corresponding to the fixation point physical coordinate; and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate to obtain the target fixation point coordinate.

In an exemplary embodiment of the present disclosure, the gaze point pixel coordinates include a left eye pixel coordinate and a right eye pixel coordinate; the target coordinate determination unit includes a first coordinate conversion subunit configured to: determining a central point pixel coordinate corresponding to the human eyes according to the left eye pixel coordinate and the right eye pixel coordinate; and performing the physical coordinate conversion on the pixel coordinate of the central point to obtain the physical coordinate of the fixation point.

In an exemplary embodiment of the present disclosure, the target coordinate determination unit includes a second coordinate conversion subunit operable to: acquiring an image equipment coordinate system corresponding to image acquisition equipment; determining a first transformation matrix matching the image device coordinate system; and performing three-dimensional coordinate conversion on the physical coordinates of the fixation point based on the first conversion matrix to obtain the initial three-dimensional fixation point coordinates.

In an exemplary embodiment of the present disclosure, the target device includes a driving device; the target coordinate determination unit includes a third coordinate conversion subunit operable to: acquiring a driving equipment coordinate system corresponding to the driving equipment; determining a second conversion matrix corresponding to the coordinate system of the driving equipment, wherein the second conversion matrix is determined based on the physical size information of the driving equipment and the equipment calibration information of the image acquisition equipment; and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate based on the second conversion matrix to obtain the target fixation point coordinate.

In an exemplary embodiment of the present disclosure, the image adjustment parameters include a translation matrix and a rotation matrix, and the image adjustment module includes a first image adjustment unit for: acquiring image display parameters of the augmented reality image, wherein the image display parameters comprise a current display coordinate and a current rotation angle; and carrying out translation processing on the current display coordinates of the augmented reality image based on the translation matrix, and adjusting the current rotation angle of the augmented reality image based on the rotation matrix so as to adjust the display position of the augmented reality image.

In an exemplary embodiment of the disclosure, the image adjustment parameters include a translation matrix and a rotation matrix, and the image adjustment module includes a second image adjustment unit to: acquiring reflector parameters of an image reflector, wherein the image reflector is used for reflecting and displaying the augmented reality image; adjusting the reflector parameters according to the translation matrix and the rotation matrix so as to adjust the position of the image reflector; and displaying the augmented reality image based on the image reflector after the position adjustment processing so as to adjust the display position of the augmented reality image.

According to a fourth aspect of the present disclosure, there is provided an electronic apparatus comprising: a processor; and a memory having computer readable instructions stored thereon which, when executed by the processor, implement the image processing method according to any one of the above.

According to a fifth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements an image processing method according to any one of the above.

The technical scheme provided by the disclosure can comprise the following beneficial effects:

in the image processing method in the exemplary embodiment of the present disclosure, on one hand, it is detected that the current gazing point coordinate has a position change, and the display position of the augmented reality image is adjusted according to the generated image adjustment parameter so as to fall within the visible range of human eyes, without manual operation. On the other hand, because the image adjustment parameters are generated according to the fixation point coordinates after the position changes, the position matching relation between the augmented reality image and human eyes can be matched again, the augmented reality image is ensured to be in the visual range of the human eyes, and the virtual-real matching precision is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty. In the drawings:

fig. 1 schematically shows a flow chart of an image processing method according to an exemplary embodiment of the present disclosure;

fig. 2 schematically illustrates an overall flow diagram for adjusting an augmented reality image according to a human eye according to an exemplary embodiment of the present disclosure;

FIG. 3 schematically illustrates a schematic view of a gaze location of a driver, according to an exemplary embodiment of the present disclosure;

fig. 4 schematically illustrates an effect diagram of augmented reality image adjustment according to an exemplary embodiment of the present disclosure;

FIG. 5 schematically illustrates a block diagram of an image processing system according to an exemplary embodiment of the present disclosure;

fig. 6 schematically shows a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure;

FIG. 7 schematically shows a block diagram of an electronic device according to an exemplary embodiment of the present disclosure;

fig. 8 schematically illustrates a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and a repetitive description thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the embodiments of the disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in the form of software, or in one or more software-hardened modules, or in different networks and/or processor devices and/or microcontroller devices.

In the existing AR-HUD implementation scheme, the problem that the virtual image surface is incompletely displayed is easily caused due to the fact that the angle of view of the AR-HUD is small and the sitting posture of a driver is changed in the driving process, and meanwhile the AR-HUD can also have the problem that the virtual image surface and the real image surface do not correspond to each other, so that deviation between the virtual image and a real-scene target is caused, unsafe factors are brought to the driving process, and user experience is reduced.

Currently, the mainstream operation for the AR-HUD display effect is still a manual adjustment method, however, the manual adjustment method has two problems: (1) the operation is time-consuming and labor-consuming; (2) The threshold of the operation method for the user is high, and the adjustment to the correct range is difficult.

Based on this, the present disclosure proposes an image processing method, an image processing system, an image processing apparatus, an electronic device, and a computer-readable storage medium.

In this context, it is understood that the terms referred to, for example, augmented Reality (AR), are a technique of calculating the position and angle of the camera image in real time and adding the corresponding image, integrating real world information and virtual world information "seamlessly", the objective of augmented reality being to fit the virtual world around the real world on a screen and interact with it. Head-up display (HUD), also known as head-up display system, refers to a multifunctional instrument panel that is operated blindly, centered on the vehicle driver.

In the present exemplary embodiment, first, an image processing method is provided, where the image processing method of the present disclosure may be implemented by a server, and the method of the present disclosure may also be implemented by a terminal device, where the terminal described in the present disclosure may include a mobile terminal such as a mobile phone, a tablet computer, a notebook computer, a palm computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a navigation device, a wearable device, a smart band, a pedometer, and a fixed terminal such as a desktop computer. Fig. 1 schematically illustrates a schematic diagram of a flow of an image processing method according to some embodiments of the present disclosure. Referring to fig. 1, the image processing method may include the steps of:

step S110, determining the current fixation point coordinate, wherein the current fixation point coordinate is used for indicating the coordinate of the current fixation position of human eyes.

According to some exemplary embodiments of the present disclosure, the current gaze point coordinates may be position coordinates to which the gaze point of the driver currently corresponds; wherein the point of regard may be a point in the visual perception process at which the line of sight is directed towards the object.

Augmented reality technology has been applied to a variety of real scenes, such as the AR-HUD system in a vehicle driving scene. The present disclosure will be described taking an AR-HUD system as an example. In the AR-HUD system, the driving device may first locate the current gaze point coordinates of the human eyes in order to provide real-time traffic information, and then display a real-time traffic image at a suitable position according to the current gaze point coordinates. Determining the coordinates of the current fixation point through a data file containing human eye information; for example, a data file containing eye information may include, but is not limited to, a facial image, a depth map, or other type of eye map.

Taking the determination of the coordinates of the current fixation point based on the face image as an example, the specific process is as follows: and acquiring a face image through image acquisition equipment to acquire the face image of the driver. After the face image is obtained, face feature extraction is carried out on the face image to determine the human eye information of the driver; the eye information may include coordinates of a current gazing point of the eyes, that is, position coordinates of the current gazing point of the eyes.

Step S120, in response to the position change instruction corresponding to the detected current fixation point coordinate, coordinate conversion is carried out on the current fixation point coordinate to obtain a target fixation point coordinate of human eyes relative to the target device.

According to some exemplary embodiments of the present disclosure, the position change instruction may be an instruction generated based on an event that changes a position coordinate of the gazing point. The coordinate conversion may be a process of performing coordinate conversion on the current gazing point coordinate. The target device may be a device where the user is currently located, for example, the target device may be a driving device or a gaming device. The target fixation point coordinate may be a target fixation point coordinate of the human eye with respect to the target device generated after coordinate conversion is performed on the current fixation point coordinate.

After the current fixation point coordinates of the human eyes are obtained, whether the positions of the human eyes are changed or not can be judged according to the current fixation point coordinates. For example, in the driving process, the coordinates of the fixation point may change due to the situations such as sitting posture adjustment, and at this time, the visible range of the human eyes will also change correspondingly, but the display position of the image cannot be dynamically adjusted according to the visible range of the human eyes, so that the human eyes cannot see the complete real-time road condition image.

In order to solve the problem that a driver needs to manually adjust the display position of the real-time road condition image in the driving process, the current fixation point coordinates of human eyes can be detected in real time in the driving process. For example, whether a position change instruction of the current gazing point coordinate is received is detected in real time. And if a position change instruction of the current fixation point coordinate is received, responding to the position change instruction, performing coordinate conversion processing on the current fixation point coordinate, and converting the position pixel coordinate of the human eye into a target fixation point coordinate of the human eye relative to a target equipment coordinate system.

In this embodiment, the target device may be determined according to a specific application scenario. For example, in a vehicle driving scenario, the target device may be a driving device driven by the driver; in a video game scenario at a casino, the target device may be a simulated driving device used by the player; in a virtual game scenario, the target device may be a virtual driving device or the like driven by a virtual player. The present disclosure does not impose any particular limitation on the specific type of target device.

Step S130, acquiring the augmented reality image, and determining the image adjustment parameters of the augmented reality image according to the target fixation point coordinates.

According to some example embodiments of the present disclosure, the augmented reality image may be an image generated by integrating real world information with virtual world information. The image adjustment parameter may be a parameter used for adjusting the display position of the augmented reality image.

In a vehicle driving scene, in order to provide real-time road condition information to a driver, a real-time road condition image may be provided for the driver in real time based on a windshield, and the real-time road condition image may be an image generated based on an augmented reality technology, for example, an augmented reality image may be generated by combining various information such as a real-time path and vehicle driving prompt information.

Since in the foregoing step, after coordinate conversion is performed on the current gaze point coordinates of the human eye, corresponding target gaze point coordinates are generated, which may be gaze point position coordinates in the coordinate system of the target device. Accordingly, an image adjustment parameter for adjusting the display position of the augmented reality image may be determined according to the target gaze point coordinates to adjust the display position of the augmented reality image.

And step S140, carrying out position adjustment processing on the augmented reality image based on the image adjustment parameters so as to enable the augmented reality image to be in the visible range of human eyes.

According to some exemplary embodiments of the present disclosure, the position adjustment process may be a process of adjusting a display position of the augmented reality image. The visible range of the human eye may be the range of the environment visible to the human eye.

Optionally, the augmented reality image is subjected to position adjustment processing based on the image adjustment parameter, so that the augmented reality image is entirely within the visible range of human eyes.

After the image adjustment parameters are determined, the display position of the augmented reality image can be automatically adjusted based on the image adjustment parameters, and the display parameters of the AR-HUD system do not need to be manually adjusted by a driver. The purpose of the image adjustment is to generate an augmented reality image that can be within the visible range of the human eye. And moreover, the position of the augmented reality image is adjusted, the virtual (virtual image generated according to road conditions) and real (human eyes and actual paths) corresponding relation can be matched again, and the AR-HUD system is ensured to be in a normal working state.

According to the image processing method in the present exemplary embodiment, on one hand, it is detected that the current gazing point coordinate has a position change, and the display position of the augmented reality image is adjusted according to the generated image adjustment parameter so as to fall within the visible range of human eyes, without manual operation. On the other hand, the image adjusting parameters are generated according to the fixation point coordinates after the position changes, so that the position matching relationship between the augmented reality image and human eyes can be matched again, the augmented reality image is ensured to be in the visual range of the human eyes, and the virtual-real matching precision is improved.

Next, the image processing method in the present exemplary embodiment will be further explained.

In an exemplary embodiment of the present disclosure, determining the current gaze point coordinates comprises: acquiring a face image through image acquisition equipment; acquiring a pre-trained facial feature extraction model, wherein the facial feature extraction model is obtained by training a facial image training set; and extracting the features of the face image through a face feature extraction model to obtain corresponding eye information, wherein the eye information comprises the coordinates of the current fixation point.

The face image may be an image of the face of the user acquired by the image acquisition device. The image capturing device may be a device for capturing an image of the face of the user. The facial feature extraction model may be a network model for extracting facial features contained in the facial image. The facial image training set may be an image training set used to train a facial feature extraction model. The human eye information may be related information of the user's eyes.

Referring to fig. 2, fig. 2 schematically shows an overall flowchart for adjusting an augmented reality image according to a human eye according to an exemplary embodiment of the present disclosure. In step S201, the process is initialized. In the initialization process, the equipment is mainly subjected to self-checking and other processing, and if the equipment is the driving equipment, whether the driving equipment and other equipment in the driving equipment, such as image acquisition equipment, image display equipment and the like, are in a normal working state is checked.

In step S202, an image of the face of the driver is acquired. Acquiring a face image of a driver through image acquisition equipment; the image acquisition equipment can be the camera, gathers driver's face image through the camera, can select for use infrared camera if the camera for the various light condition of adaptation daytime and evening, and then carries out facial feature positioning according to the face image that acquires.

In step S203, facial features are located. In this embodiment, a facial feature extraction model obtained based on deep learning training may be used to detect and locate facial features in a facial image. The facial feature extraction model can be obtained by training based on a facial image training set, and the facial image training set can comprise normal facial images or partially covered facial images, such as facial images with glasses, facial images with masks and the like. The human eye is identified from the face image and the subsequent processing steps are performed.

In step S204, the current gaze point coordinates are located, i.e. the gaze point pixel coordinates of the human eye are determined. In step S203, the human eyes are positioned on the face of the person, and the human eye information is determined. The eye information may include the gaze point pixel position, that is, the current gaze point coordinates of the eyes are determined based on the eye information, and the current pixel position coordinates of the left and right gaze points of the person are finally obtained.

Referring to fig. 3, fig. 3 schematically illustrates a schematic view of a gaze location of a driver according to an exemplary embodiment of the present disclosure. In the driving process, a corresponding driving device coordinate system can be established based on the driving device, and the coordinate system comprises a horizontal direction, a vertical direction and a depth direction; the horizontal direction may be a left-right direction with respect to a position where the driver is located, the vertical direction may be a direction in which the driver is perpendicular to the driving device (or the ground), and the depth direction may be a direction in which the driving device moves forward or backward during driving.

In an exemplary embodiment of the present disclosure, reference gaze point coordinates corresponding to a pre-stored reference position of a human eye are obtained; comparing the current fixation point coordinate with the reference fixation point coordinate to obtain a coordinate change difference value corresponding to the current fixation point coordinate; and if the coordinate change difference value is greater than or equal to a pre-configured coordinate change threshold value, generating a position change instruction.

The reference position of the human eye may be a reference position for determining whether the human eye changes. The reference gaze point coordinates may be pixel coordinates corresponding to a reference location of the human eye. The coordinate change difference may be a pixel difference calculated by comparing the current gazing point coordinate with the reference gazing point coordinate. The coordinate change threshold value can be a preset coordinate change reference value, and the coordinate change difference value is compared with the coordinate change threshold value, so that whether the position of human eyes is changed or not can be judged.

With continued reference to fig. 2, in step S205, it is determined whether the gaze location has changed. The determining step may compare the current gazing point position (i.e., the current gazing point coordinates) with the stored reference position coordinates to determine whether a change has occurred. Specifically, a pre-stored reference position of the human eye may be obtained, and the reference position of the human eye may be a pre-stored pixel position of the human eye detected at a detection time. After the reference position of the human eye is obtained, the reference fixation point coordinate corresponding to the reference position of the human eye can be determined, and the determined current fixation point coordinate is compared with the reference fixation point coordinate to obtain a difference value between the two coordinates, wherein the difference value is used as a coordinate change difference value.

Referring to fig. 4, fig. 4 schematically illustrates an effect diagram of augmented reality image adjustment according to an exemplary embodiment of the present disclosure. In the driving process, the position of a fixation point of a driver may be changed due to the change of the sitting posture of the driver, and at the moment, an augmented reality image presented at the last moment is still displayed at the original image position, so that the situation that the image cannot be completely within the visual range of human eyes may be caused; in addition, if the driving apparatus is replaced with a new driver, the above-mentioned problem may also be caused due to the difference in height of the driver. In order to solve the above problem, it is necessary to determine a coordinate change difference of human eyes, determine whether a position of a gaze point of a driver changes, and determine whether to adjust a position of an augmented reality image according to a determination result.

In this embodiment, in order to implement a smooth transition of the augmented reality image, when determining whether the current gazing point coordinate changes, a coordinate change threshold for comparing with a coordinate change difference may be configured in advance, where the coordinate change threshold is used to determine whether the position of the human eye has changed. If the coordinate change difference is greater than or equal to the pre-configured coordinate change threshold, indicating that the range of the eye position change is large, the gaze position may be considered to have changed at this time, and therefore a corresponding position change command may be generated. If the coordinate change difference is smaller than the pre-configured coordinate change threshold, it is considered that the human eyes are adjusted in a normal sitting posture during driving, and it can be considered that the current positions of the human eyes are not changed, at this time, the processing flow may return to step S202 to perform the operation of re-acquiring the face image.

It should be noted that, in a vehicle driving scene, a scene causing a change in the position of the gaze point of the human eye may include: the difference of different heights of drivers causes the position of the fixation point to change; changes in the driver's sitting posture during driving cause changes in the gaze location, and so on.

In addition, the image processing method of the present disclosure may also be applied to other scenes, for example, a simulated driving game in a game hall, a virtual driving game in a virtual game, and the like, and in the simulated driving game and the virtual driving game, a scene in which the position of the gazing point changes is also involved, which is not described in detail in the present disclosure.

In an exemplary embodiment of the present disclosure, the performing coordinate transformation on the current gaze point coordinate to obtain a target gaze point coordinate of the human eye relative to the target device includes: performing physical coordinate conversion on the fixation point pixel coordinate to obtain a fixation point physical coordinate corresponding to the fixation point pixel coordinate; performing three-dimensional coordinate conversion on the physical coordinates of the fixation points to obtain initial three-dimensional fixation point coordinates corresponding to the physical coordinates of the fixation points; and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate to obtain a target fixation point coordinate.

The physical coordinate conversion may be a conversion process of converting the gaze point pixel coordinates of the human eye into gaze point physical coordinates. The gaze point physical coordinates may be coordinates representing the position of the human eye by physical dimensions of the image capturing device, for example, the gaze point physical coordinates may refer to converting gaze point pixel coordinates into two-dimensional physical coordinates on a light sensing element in a camera corresponding to the human eye. The three-dimensional coordinate conversion may be a conversion process of converting the two-dimensional gaze point physical coordinates into three-dimensional coordinates with respect to the camera. The initial three-dimensional fixation point coordinate is as follows: and (4) performing three-dimensional coordinate conversion on the physical coordinates of the fixation point to generate the fixation point coordinates. The device mapping coordinate conversion may be a coordinate conversion process of converting the initial three-dimensional gaze point coordinates with respect to the image device coordinate system to target gaze point coordinates with respect to the driving device coordinate system. The target gaze point coordinates may be position coordinates of the human eye relative to the coordinate system of the driving device.

The current fixation point coordinate extracted by the facial feature extraction model can be a fixation point pixel coordinate of human eyes, and after the current fixation point coordinate is obtained, coordinate conversion can be carried out on the current fixation point coordinate to obtain a target fixation point coordinate of the human eyes relative to target equipment. The method comprises the following specific steps: for the obtained fixation point pixel coordinates, physical coordinate conversion can be performed on the fixation point pixel coordinates firstly, the pixel coordinates are converted into corresponding physical coordinates, and then the corresponding fixation point physical coordinates are obtained, wherein the fixation point physical coordinates can be position coordinates of human eyes which are adaptive to the physical size of the image acquisition equipment, and the fixation point physical coordinates are two-dimensional coordinates.

After the physical coordinates of the fixation point are obtained, three-dimensional coordinate conversion is carried out on the physical coordinates of the fixation point to obtain corresponding initial three-dimensional fixation point coordinates, and the initial three-dimensional fixation point coordinates can be three-dimensional position coordinates of human eyes relative to the image acquisition equipment. For the obtained initial three-dimensional fixation point coordinates, device mapping coordinate conversion can be performed on the initial three-dimensional fixation point coordinates, and the initial three-dimensional fixation point coordinates are converted into target fixation point coordinates relative to a coordinate system of target devices, so that image adjustment parameters for adjusting the augmented reality image can be determined according to the obtained target fixation point coordinates.

In an exemplary embodiment of the present disclosure, the gaze point pixel coordinates comprise left eye pixel coordinates and right eye pixel coordinates; the method for performing physical coordinate conversion on the fixation point pixel coordinate to obtain the fixation point physical coordinate corresponding to the fixation point pixel coordinate comprises the following steps: determining a central point pixel coordinate corresponding to human eyes according to the left eye pixel coordinate and the right eye pixel coordinate; and carrying out physical coordinate conversion on the pixel coordinate of the central point to obtain the physical coordinate of the fixation point.

The left-eye pixel coordinate may be a pixel coordinate corresponding to the left eye of the human. The right eye pixel coordinates may be pixel coordinates corresponding to a right eye of the person. The center-point pixel coordinate may be a pixel coordinate of the center point determined from the left-eye pixel coordinate and the right-eye pixel coordinate.

With continued reference to fig. 2, in step S206, the pixel coordinates of the center points of the left and right gazing points are acquired. The current fixation point coordinates of the human eyes acquired in step S204 are pixel coordinates of the human eyes, including left-eye pixel coordinates and right-eye pixel coordinates of the human eyes, and the left-eye pixel coordinates are (x 1, y 1), and the right-eye pixel coordinates are (x 2, y 2). Then, the pixel coordinate of the center point of the human eye is calculated according to the pixel coordinate of the left eye and the pixel coordinate of the right eye, and the pixel coordinate of the center point can be the coordinate of the center point of the connecting line of the left eye and the right eye, namely the pixel coordinate of the center point is ((x 1+ x 2)/2, (y 1+ y 2)/2). Before step S207 is executed, the physical coordinate conversion is performed on the central pixel coordinate, so as to obtain the corresponding gaze point physical coordinate, and perform a subsequent coordinate conversion process based on the gaze point physical coordinate.

In an exemplary embodiment of the present disclosure, performing three-dimensional coordinate conversion on a physical coordinate of a gaze point to obtain an initial three-dimensional gaze point coordinate corresponding to the physical coordinate of the gaze point includes: acquiring an image equipment coordinate system corresponding to image acquisition equipment; determining a first conversion matrix matched with a coordinate system of the image equipment; and performing three-dimensional coordinate conversion on the physical coordinates of the fixation point based on the first conversion matrix to obtain initial three-dimensional fixation point coordinates.

Wherein, the image device coordinate system may be a coordinate system established based on the image capturing device. The first transformation matrix may be a coordinate transformation matrix generated based on relevant device parameters of the image acquisition device.

With continued reference to fig. 2, in step S207, the gaze point physical coordinates are converted into a camera coordinate system. Specifically, an image device coordinate system corresponding to the image capturing device is obtained first, and the image device coordinate system is based on a coordinate system established by the image capturing device, for example, the image device coordinate system may be a coordinate system established by using the light spot as an origin, and using an X axis and a Y axis parallel to an X axis and a Y axis of the image coordinate system, respectively, and using an optical axis of the camera as a Z axis. After the coordinate system of the image device is obtained, a first transformation matrix matched with the coordinate system can be obtained, and the first transformation matrix can be composed of an internal parameter matrix, an external parameter matrix, a distortion matrix and the like. The first conversion matrix may be generated based on calibration results of the image acquisition device.

After the first conversion matrix is obtained, three-dimensional coordinate conversion may be performed on the physical coordinates of the gaze point based on the first conversion matrix to generate corresponding initial three-dimensional gaze point coordinates, which may be three-dimensional position coordinates of the human eye relative to the coordinate system of the image device.

In an exemplary embodiment of the present disclosure, the target device includes a driving device; performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate to obtain a target fixation point coordinate, wherein the method comprises the following steps: acquiring a driving equipment coordinate system corresponding to driving equipment; determining a second conversion matrix corresponding to the coordinate system of the driving equipment, wherein the second conversion matrix is determined based on the physical size information of the driving equipment and the equipment calibration information of the image acquisition equipment; and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate based on the second conversion matrix to obtain a target fixation point coordinate.

Wherein the driving device may be a device used by a user in a vehicle driving scenario. The steering device coordinate system may be a coordinate system generated relative to the steering device. The second transformation matrix may be a coordinate transformation matrix generated based on the physical size information of the driving device and the calibration device parameters of the image capturing device. The physical size information of the driving device may include an outer size of the driving device, a cab parameter, a driving speed of the driving device, a driving direction, and the like. The device calibration information of the image capturing device may include internal parameters, external parameters, distortion parameters, or the like of the image capturing device.

With continued reference to fig. 2, in step S208, the initial three-dimensional gaze point coordinates are converted to the coordinate system of the steering device. And converting the three-dimensional coordinate of the gazing point position in the coordinate system of the image acquisition equipment into the coordinate system of the driving equipment. Before the device mapping coordinate conversion is performed, a second conversion matrix may be obtained, and the second conversion matrix may be generated based on the physical size data of the driving device and the calibration data of the image capture device. After the second conversion matrix is obtained, device mapping coordinate conversion is carried out on the initial three-dimensional fixation point coordinate based on the second conversion matrix, and a three-dimensional position coordinate of human eyes relative to a coordinate system of the driving device, namely a target fixation point coordinate is obtained.

After obtaining the target gaze point coordinates, in step S209, the target gaze point coordinates are sent to the central processing unit of the driving device, so that the central processing unit performs position adjustment on the presented augmented reality image based on the target gaze point coordinates.

In an exemplary embodiment of the present disclosure, the image adjustment parameters include a translation matrix and a rotation matrix, and the performing the position adjustment process on the augmented reality image based on the image adjustment parameters includes: acquiring image display parameters of an augmented reality image, wherein the image display parameters comprise current display coordinates and a current rotation angle; and carrying out translation processing on the current display coordinates of the augmented reality image based on the translation matrix, and adjusting the current rotation angle of the augmented reality image based on the rotation matrix so as to adjust the display position of the augmented reality image.

The translation matrix may be a matrix composed of translation amounts of the augmented reality image in a horizontal direction, a vertical direction, and a depth direction. The rotation matrix may be a matrix composed of rotation angles of the augmented reality image in a horizontal direction, a vertical direction, and a depth direction. The image display parameters may be used to control parameters employed for augmented reality image display. The current display coordinates may be display coordinates of the augmented reality image in a horizontal direction, a vertical direction, and a depth direction at present. The current rotation angle may be a rotation angle of the augmented reality image in a horizontal direction, a vertical direction, and a depth direction at present.

With continued reference to fig. 2, in step S210, image adjustment parameters are calculated. When image adjustment parameters are calculated, six parameters such as translation vectors and rotation vectors of the movement of the augmented reality image can be calculated according to the coordinates of the target fixation point and by combining the optical design parameters of the AR-HUD, and then a translation matrix and a rotation matrix can be obtained. After the translation matrix and the rotation matrix are obtained, image display parameters of the augmented reality image can be obtained, namely the current display coordinates and the current rotation angle of the augmented reality image when the augmented reality image is displayed based on the driving equipment; the current display coordinates may include position coordinates of the image in a horizontal direction, a vertical direction, and a depth direction; the current rotation angle may include rotation angles of the image in a horizontal direction, a vertical direction, and a depth direction.

In step S211, an image adjustment step is performed. In step S212, an image adjustment is performed on the augmented reality image by adopting a pixel adjustment method. The pixel adjusting method can directly adjust the pixel position and the rotation angle of the augmented reality image on the projection surface of the AR-HUD light machine so as to complete the adjustment of the image display position.

Specifically, the current display coordinates of the augmented reality image may be translated based on the translation matrix; since the translation matrix includes the moving values of the image in the horizontal direction, the vertical direction and the depth direction, the specific position of the augmented reality image in the three directions can be adjusted based on the translation matrix. In addition, the current rotation angle of the augmented reality image may be adjusted based on a rotation matrix, and the augmented reality image may be deflected in the horizontal direction, the vertical direction, and the depth direction by a rotation angle matching the human eyes based on the rotation matrix. With continued reference to fig. 4, through the above steps, the adjustment of the display position for the augmented reality image is finally completed, so that the image is in the visible range of human eyes.

In an exemplary embodiment of the present disclosure, performing a position adjustment process on an augmented reality image based on an image adjustment parameter includes: acquiring reflector parameters of an image reflector, wherein the image reflector is used for reflecting and displaying an augmented reality image; adjusting the parameters of the reflector according to the translation matrix and the rotation matrix so as to adjust the position of the image reflector; and displaying the augmented reality image based on the image mirror after the position adjustment processing so as to adjust the display position of the augmented reality image.

Wherein the image mirror may be to reflect the augmented reality image such that the augmented reality image is at a specific location of the target device. The mirror parameter may be a device parameter associated with the mirror.

With continued reference to fig. 2, in step S213: and image adjustment is carried out on the augmented reality image by adopting the overturning of the reflecting mirror. The AR-HUD augmented reality image adjusting method is still realized by adopting a method of turning a reflector. In the implementation scheme of the mirror turning, the mirror parameters are adjusted by using a translation matrix and a rotation matrix, so that the driving motor controls the turning of the mirror according to the adjusted mirror parameters.

Similar to the pixel adjustment manner, in the mirror flipping scheme, the motors may be driven to control the moving distances of the image mirror in the horizontal direction, the vertical direction, and the depth direction based on the translation matrix, and the motors may be driven to adjust the rotation angles of the image mirror in the horizontal direction, the vertical direction, and the depth direction based on the rotation matrix. After the position of the image reflector is adjusted, the augmented reality image can be reflected and displayed at the corresponding position of the driving equipment based on the image reflector so as to adjust the display position of the augmented reality image, at the moment, the augmented reality image after position adjustment is within the visual range of human eyes, the virtual-real corresponding relation among the human eyes, the real-time road condition and the generated virtual driving prompt information is matched again, and the normal working state of the AR-HUD system is ensured.

It will be readily appreciated by those skilled in the art that other equivalent-effect image adjustment methods may also be employed to achieve a positional adjustment of the augmented reality image to bring the augmented reality image within the visible range of the human eye in other exemplary embodiments of the present disclosure. The present disclosure is not limited thereto in any particular way.

It should be noted that the terms "first", "second", etc. are used in this disclosure only for distinguishing different transformation matrices, and should not set any limit to this disclosure.

In summary, the image processing method of the present disclosure determines a current gaze point coordinate, where the current gaze point coordinate is used to indicate a coordinate of a current gaze position of human eyes; performing coordinate conversion on the current fixation point coordinate in response to the detected position change instruction of the current fixation point coordinate to obtain a target fixation point coordinate of human eyes relative to target equipment; acquiring an augmented reality image, and determining image adjustment parameters of the augmented reality image according to the coordinates of the target fixation point; and carrying out position adjustment processing on the augmented reality image based on the image adjustment parameters so as to enable the augmented reality image to be in the visible range of human eyes. On one hand, the position change of the current fixation point coordinate is detected, and the display position of the augmented reality image is adjusted according to the generated image adjusting parameter, so that the augmented reality image falls into the visual range of human eyes, and manual operation is not needed. On the other hand, the image adjusting parameters are generated according to the fixation point coordinates after the position changes, so that the position matching relationship between the augmented reality image and human eyes can be matched again, the augmented reality image is ensured to be in the visual range of the human eyes, and the virtual-real matching precision is improved. On the other hand, the normal working state of the AR-HUD system can be ensured because the virtual-real corresponding relation among the human eyes, the real-time road conditions and the generated virtual driving prompt information can be adjusted in real time according to the real-time position change of the human eyes.

It is noted that although the steps of the methods of the present invention are depicted in the drawings in a particular order, this does not require or imply that the steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.

Further, the present disclosure also provides an image processing system. Referring to fig. 5, the image processing system 500 may include: the image acquisition device 510 is configured to acquire a face image, and determine a current fixation point coordinate corresponding to the human eye based on the face image; the coordinate conversion module 520 is configured to perform coordinate conversion processing on the current fixation point coordinate to obtain a target fixation point coordinate of the human eye relative to the target device; a transmission module 530 for transmitting the target fixation point coordinates to the image adjustment module; an image position adjustment module 540, configured to determine an image adjustment parameter of the augmented reality image based on the target gaze point coordinate; and carrying out position adjustment processing on the augmented reality image based on the image adjustment parameters so as to enable the augmented reality image to be in the visible range of human eyes. Based on the position adjustment of the augmented reality image by the image processing system 500, the working state of the AR-HUD system is ensured to be normal.

Further, in the present exemplary embodiment, an image processing apparatus is also provided. Referring to fig. 6, the image processing apparatus 600 may include: a current coordinate determination module 610, a target coordinate determination module 620, an adjustment parameter determination module 630, and an image adjustment module 640.

Specifically, the current coordinate determining module 610 is configured to determine a current gaze point coordinate, where the current gaze point coordinate is used to indicate a coordinate of a current gaze position of human eyes; the target coordinate determination module 620 is configured to perform coordinate conversion on the current gaze point coordinate in response to detecting the position change instruction corresponding to the current gaze point coordinate, so as to obtain a target gaze point coordinate of the human eye relative to the target device; an adjustment parameter determining module 630, configured to obtain an augmented reality image, and determine an image adjustment parameter of the augmented reality image according to the target gaze point coordinate; and the image adjusting module 640 is configured to perform position adjustment processing on the augmented reality image based on the image adjustment parameter, so that the augmented reality image is entirely within the visible range of human eyes.

In an exemplary embodiment of the present disclosure, the target coordinate determination module 620 includes a target coordinate determination unit for: acquiring a face image through image acquisition equipment; acquiring a pre-trained facial feature extraction model, wherein the facial feature extraction model is obtained by training a facial image training set; and extracting the features of the face image through a face feature extraction model to obtain corresponding eye information, wherein the eye information comprises the coordinates of the current fixation point.

In an exemplary embodiment of the present disclosure, the image processing apparatus 600 further includes a coordinate change detection module, configured to obtain a reference fixation point coordinate corresponding to a pre-stored reference position of the human eye; comparing the current fixation point coordinate with the reference fixation point coordinate to obtain a coordinate change difference value corresponding to the current fixation point coordinate; and if the coordinate change difference value is larger than or equal to a pre-configured coordinate change threshold value, generating a position change instruction.

In an exemplary embodiment of the present disclosure, the current gaze point coordinates comprise gaze point pixel coordinates, and the target coordinate determination module 620 comprises a target coordinate determination unit to: performing physical coordinate conversion on the fixation point pixel coordinate to obtain a fixation point physical coordinate corresponding to the fixation point pixel coordinate; performing three-dimensional coordinate conversion on the physical coordinates of the fixation points to obtain initial three-dimensional fixation point coordinates corresponding to the physical coordinates of the fixation points; and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate to obtain a target fixation point coordinate.

In an exemplary embodiment of the present disclosure, the gaze point pixel coordinates comprise left eye pixel coordinates and right eye pixel coordinates; the target coordinate determination unit includes a first coordinate conversion subunit operable to: determining a central point pixel coordinate corresponding to human eyes according to the left eye pixel coordinate and the right eye pixel coordinate; and carrying out physical coordinate conversion on the pixel coordinate of the central point to obtain the physical coordinate of the fixation point.

In an exemplary embodiment of the present disclosure, the target coordinate determination unit includes a second coordinate conversion subunit operable to: acquiring an image equipment coordinate system corresponding to image acquisition equipment; determining a first conversion matrix matched with a coordinate system of the image equipment; and performing three-dimensional coordinate conversion on the physical coordinates of the fixation point based on the first conversion matrix to obtain initial three-dimensional fixation point coordinates.

In an exemplary embodiment of the present disclosure, the target device includes a driving device; the target coordinate determination unit includes a third coordinate conversion subunit operable to: acquiring a driving equipment coordinate system corresponding to driving equipment; determining a second conversion matrix corresponding to the coordinate system of the driving equipment, wherein the second conversion matrix is determined based on the physical size information of the driving equipment and the equipment calibration information of the image acquisition equipment; and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate based on the second conversion matrix to obtain a target fixation point coordinate.

In an exemplary embodiment of the present disclosure, the image adjustment parameters include a translation matrix and a rotation matrix, and the image adjustment module 640 includes a first image adjustment unit for: acquiring image display parameters of an augmented reality image; the image display parameters comprise current display coordinates and a current rotation angle; carrying out translation processing on the current display coordinate of the augmented reality image based on the translation matrix; and adjusting the current rotation angle of the augmented reality image based on the rotation matrix so as to adjust the display position of the augmented reality image.

In an exemplary embodiment of the present disclosure, the image adjustment parameters include a translation matrix and a rotation matrix, and the image adjustment module 640 includes a second image adjustment unit for: acquiring reflector parameters of an image reflector, wherein the image reflector is used for reflecting and displaying an augmented reality image; adjusting the parameters of the reflector according to the translation matrix and the rotation matrix so as to adjust the position of the image reflector; and displaying the augmented reality image based on the image mirror after the position adjustment processing so as to adjust the display position of the augmented reality image.

The specific details of the virtual modules of the image processing apparatuses are already described in detail in the corresponding image processing methods, and therefore are not described herein again.

It should be noted that although in the above detailed description several modules or units of the image processing apparatus are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Accordingly, various aspects of the present invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

An electronic device 700 according to such an embodiment of the disclosure is described below with reference to fig. 7. The electronic device 700 shown in fig. 7 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 7, electronic device 700 is embodied in the form of a general purpose computing device. The components of the electronic device 700 may include, but are not limited to: the at least one processing unit 710, the at least one memory unit 720, a bus 730 connecting different system components (including the memory unit 720 and the processing unit 710), and a display unit 740.

Wherein the memory unit stores program code that is executable by the processing unit 710 to cause the processing unit 710 to perform steps according to various exemplary embodiments of the present disclosure as described in the "exemplary methods" section above in this specification.

The memory unit 720 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM) 721 and/or a cache memory unit 722, and may further include a read only memory unit (ROM) 723.

The memory unit 720 may include a program/utility 724 having a set (at least one) of program modules 725, such program modules 725 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may comprise an implementation of a network environment.

Bus 730 may represent one or more of any of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 700 may also communicate with one or more external devices 770 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 700, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 700 to communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interfaces 750. Also, the electronic device 700 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 760. As shown, the network adapter 760 communicates with the other modules of the electronic device 700 via the bus 730. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, to name a few.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above-mentioned "exemplary methods" section of the present description, when said program product is run on the terminal device.

Referring to fig. 8, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In situations involving remote computing devices, the remote computing devices may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to external computing devices (e.g., through the internet using an internet service provider).

Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed, for example, synchronously or asynchronously in multiple modules.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the terms of the appended claims.

Claims

1. An image processing method, comprising:

determining the coordinates of a current fixation point, wherein the coordinates of the current fixation point are used for indicating the coordinates of the current fixation position of human eyes;

performing coordinate conversion on the current fixation point coordinate in response to the detection of the position change instruction corresponding to the current fixation point coordinate to obtain a target fixation point coordinate of the human eye relative to target equipment;

acquiring an augmented reality image, and determining image adjustment parameters of the augmented reality image according to the target fixation point coordinates;

and carrying out position adjustment processing on the augmented reality image based on the image adjustment parameters so as to enable the augmented reality image to be in the visible range of the human eyes.

2. The method of claim 1, wherein the determining current gaze point coordinates comprises:

acquiring a face image through image acquisition equipment;

acquiring a pre-trained facial feature extraction model, wherein the facial feature extraction model is obtained by training a facial image training set;

and extracting the features of the face image through the face feature extraction model to obtain the eye information corresponding to the face image, wherein the eye information comprises the current fixation point coordinate.

3. The method according to claim 1 or 2, characterized in that the method further comprises:

acquiring a reference fixation point coordinate corresponding to a pre-stored human eye reference position;

comparing the current fixation point coordinate with the reference fixation point coordinate to obtain a coordinate change difference value corresponding to the current fixation point coordinate;

and if the coordinate change difference value is greater than or equal to a pre-configured coordinate change threshold value, generating the position change instruction.

4. The method of claim 1, wherein the current gaze point coordinates comprise gaze point pixel coordinates, and wherein the coordinate transforming the current gaze point coordinates to obtain target gaze point coordinates of the human eye relative to a target device comprises:

performing physical coordinate conversion on the fixation point pixel coordinate to obtain a fixation point physical coordinate corresponding to the fixation point pixel coordinate;

performing three-dimensional coordinate conversion on the fixation point physical coordinate to obtain an initial three-dimensional fixation point coordinate corresponding to the fixation point physical coordinate;

and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate to obtain the target fixation point coordinate.

5. The method of claim 4, wherein the gaze point pixel coordinates comprise left eye pixel coordinates and right eye pixel coordinates; the physical coordinate conversion of the fixation point pixel coordinate to obtain the fixation point physical coordinate corresponding to the fixation point pixel coordinate comprises the following steps:

determining a central point pixel coordinate corresponding to the human eyes according to the left eye pixel coordinate and the right eye pixel coordinate;

and performing the physical coordinate conversion on the pixel coordinate of the central point to obtain the physical coordinate of the fixation point.

6. The method according to claim 4, wherein the performing three-dimensional coordinate transformation on the physical coordinates of the gaze point to obtain initial three-dimensional gaze point coordinates corresponding to the physical coordinates of the gaze point comprises:

acquiring an image equipment coordinate system corresponding to image acquisition equipment;

determining a first transformation matrix matching the image device coordinate system;

and performing three-dimensional coordinate conversion on the physical coordinates of the fixation point based on the first conversion matrix to obtain the initial three-dimensional fixation point coordinates.

7. The method of claim 4, wherein the target device comprises a steering device; the performing device mapping coordinate conversion on the initial three-dimensional fixation point coordinate to obtain the target fixation point coordinate comprises:

acquiring a driving equipment coordinate system corresponding to the driving equipment;

determining a second conversion matrix corresponding to the coordinate system of the driving equipment, wherein the second conversion matrix is determined based on the physical size information of the driving equipment and the equipment calibration information of the image acquisition equipment;

and performing equipment mapping coordinate conversion on the initial three-dimensional fixation point coordinate based on the second conversion matrix to obtain the target fixation point coordinate.

8. The method according to claim 1, wherein the image adjustment parameters include a translation matrix and a rotation matrix, and the performing the position adjustment processing on the augmented reality image based on the image adjustment parameters includes:

acquiring image display parameters of the augmented reality image, wherein the image display parameters comprise a current display coordinate and a current rotation angle;

performing a panning process on current display coordinates of the augmented reality image based on the panning matrix,

and adjusting the current rotation angle of the augmented reality image based on the rotation matrix so as to adjust the display position of the augmented reality image.

9. The method according to claim 1, wherein the image adjustment parameters include a translation matrix and a rotation matrix, and the performing the position adjustment processing on the augmented reality image based on the image adjustment parameters includes:

acquiring reflector parameters of an image reflector, wherein the image reflector is used for reflecting and displaying the augmented reality image;

adjusting the reflector parameters according to the translation matrix and the rotation matrix so as to adjust the position of the image reflector;

and displaying the augmented reality image based on the image reflector after the position adjustment processing so as to adjust the display position of the augmented reality image.

10. An image processing system, comprising:

the image acquisition equipment is used for acquiring a face image and determining the current fixation point coordinate corresponding to the human eyes based on the face image;

the coordinate conversion module is used for carrying out coordinate conversion processing on the current fixation point coordinate to obtain a target fixation point coordinate of the human eye relative to target equipment;

the transmission module is used for transmitting the target fixation point coordinates to the image position adjusting module;

the image position adjusting module is used for determining image adjusting parameters of the augmented reality image based on the target fixation point coordinates; and carrying out position adjustment processing on the augmented reality image based on the image adjustment parameters so as to enable the augmented reality image to be in the visible range of the human eyes.

11. An image processing apparatus characterized by comprising:

the current coordinate determination module is used for determining the coordinates of a current fixation point, and the coordinates of the current fixation point are used for indicating the coordinates of the current fixation position of human eyes;

the target coordinate determination module is used for responding to the position change instruction corresponding to the detected current fixation point coordinate, and performing coordinate conversion on the current fixation point coordinate to obtain a target fixation point coordinate of the human eye relative to the target equipment;

the adjustment parameter determining module is used for acquiring an augmented reality image and determining image adjustment parameters of the augmented reality image according to the target fixation point coordinates;

and the image adjusting module is used for carrying out position adjustment processing on the augmented reality image based on the image adjusting parameters so as to enable the augmented reality image to be positioned in the visible range of the human eyes.

12. An electronic device, comprising:

a processor; and

a memory having stored thereon computer readable instructions which, when executed by the processor, implement the image processing method according to any one of claims 1 to 9.

13. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the image processing method according to any one of claims 1 to 9.