KR20200092197A - Image processing method, image processing apparatus, electronic device, computer program and computer readable recording medium for processing augmented reality image - Google Patents

Abstract

Disclosed is an augmented reality image processing method. The method comprise the following steps of: obtaining a first driving image having a first frame rate captured while a moving object is driving; processing the obtained first driving image to generate a second driving image having a second frame rate; analyzing the second driving image to generate an augmented reality (AR) expression image including an AR guide object to be displayed on the second driving image at a second frame rate; and synchronizing and combining the generated second driving image and the AR expression image, wherein the second frame rate may be smaller than the first frame rate, thereby displaying the AR guide object at an exact position in a real-time driving image captured while the moving object is driving.

Description

Augmented reality image processing apparatus, augmented reality image processing method, electronic device, computer program, and computer readable recording medium TECHNICAL IMAGE PROCESSING METHOD

The present invention relates to an augmented reality image processing apparatus, an augmented reality image processing method, an electronic device, a computer program, and a computer-readable recording medium, and more specifically, an augmented reality image processing apparatus for solving an expression error of an augmented reality element, It relates to augmented reality image processing methods, electronic devices, computer programs, and computer readable recording media.

Augmented Reality (AR) refers to a technique of synthesizing virtual objects or information in a real environment so that they appear as objects existing in the original environment, that is, overlaying a 3D virtual object in the real world. . Unlike conventional virtual reality (VR), which targets only virtual spaces and virtual objects, augmented reality synthesizes virtual objects on the basis of the real world and reinforces and provides additional information that is difficult to obtain using only the real world. can do.

Because of this feature, unlike virtual reality, which was limitedly applied only to fields such as games, augmented reality is applied to various fields and is provided to users.

An example is augmented reality navigation that performs driving-related guidance based on augmented reality. Such augmented reality navigation may include additional information (for example, a graphic element representing a point of interest (POI), a graphic element representing a path to a destination, etc.) on a screen containing the real world that the user is actually viewing: AR guide object) is provided by visually overlapping to enhance user convenience.

In the conventional augmented reality navigation method, a driving image photographed in real time and an AR expression image are separately generated and displayed while the vehicle is driving.

However, according to such a conventional method, there is a problem in that an AR guide object is displayed in an incorrect position in a real-time driving image containing a real world actually viewed by a user due to inconsistency in image processing results for each image.

In addition, in the conventional augmented reality navigation, there is a problem in that a screen desired by a user is not output, such as an AR guide object displayed in an incorrect location in a real-time driving image due to a camera fixation problem or camera shake.

In particular, since augmented reality navigation is a combination of real-time driving images and AR guide objects that contain the real world so that it looks like objects existing in the original environment, it is very important that information is displayed at the correct location. It was difficult to achieve the purpose, and as a result, there was a problem that not only hindered the convenience and stability of the driver, but also dropped the driver's trust in navigation.

The present invention has been devised to solve the above-mentioned problems, and an object of the present invention is an augmented reality image processing apparatus and augmented reality image processing method for displaying an AR guide object at an accurate location of a real-time driving image captured while driving a moving object, It provides an electronic device, a computer program, and a computer-readable recording medium.

In addition, an object of the present invention is to compensate for the shaking of the real-time driving image taken while driving the moving object to provide a shaking-free driving image, and at the same time, an augmented reality image processing apparatus and augmented reality image processing method for displaying an AR guide object in an accurate position It provides an electronic device, a computer program, and a computer-readable recording medium.

An augmented reality image processing method using a processor according to an embodiment of the present invention according to an embodiment of the present invention for achieving the above object is to obtain a first driving image of the first frame rate taken while driving the mobile body Step, generating a second driving image at a second frame rate by processing the obtained first driving image, and analyzing the second driving image to generate an AR (Augmented Reality) guide object to be displayed on the second driving image. Generating an AR display image containing the second frame rate, and synchronizing and combining the generated second driving image and the AR display image, wherein the second frame rate is greater than the first frame rate. It can be small.

In addition, the second frame rate may be variably determined according to the performance of the processor.

Also, the second frame rate may be variably determined according to attribute information of the AR guide object.

In addition, the second frame rate may be statistically determined in consideration of attribute information of an analysis target included in the second driving image.

In addition, the step of generating the AR display image including the AR guide object at the second frame rate may include analyzing the second driving image to determine a position in the AR display image of the AR guide object and the determined image. And generating the AR expression image in which the AR guide object is positioned in my location.

Then, extracting a feature point from the second driving image, calculating the position change value of the extracted feature point and shaking data indicating the shaking size and the shaking direction of the second driving image based on the calculated position change value It may further include the step of calculating.

In addition, the method may further include correcting a position in the AR display image of the AR guide object using the calculated shake data.

In addition, the method may further include displaying an augmented reality image generated according to a combination of the second driving image at the second frame rate and the AR display image at the second frame rate.

The method further includes determining a crop region to cut a portion of the second driving image using the calculated shake data and generating a cropped image cropped from the determined crop region of the second driving image. can do.

Then, generating a third driving image in which the size of the cropped image is enlarged to the size of the second driving image and combining the third driving image at the second frame rate and the AR display image at the second frame rate. The method may further include displaying the generated augmented reality image.

On the other hand, the augmented reality image processing apparatus according to an embodiment of the present invention for achieving the above object, the image acquisition unit for acquiring the first driving image of the first frame rate photographed while the moving object, the obtained agent A driving image generator that processes a first driving image to generate a second driving image at a second frame rate, and an AR including an Augmented Reality (AR) guide object to be displayed on the second driving image by analyzing the second driving image And an AR display image generator for generating the displayed image at the second frame rate and an image combining unit for synchronizing and combining the generated second driving image and the AR displayed image, wherein the second frame rate is the first frame. It may be less than the rate.

Also, the second frame rate may be variably determined according to the performance of the processor.

Also, the second frame rate may be variably determined according to attribute information of the AR guide object.

Also, the second frame rate may be variably determined according to attribute information of an analysis target included in the second driving image.

In addition, the AR expression image generation unit analyzes the second driving image, determines the location within the AR expression image of the AR guidance object, and positions the AR guidance image in which the AR guidance object is positioned in the determined location within the image. Can be created.

Then, the feature point is extracted from the second driving image, the position change value of the extracted feature point is calculated, and the shaking data indicating the shaking size and the shaking direction of the second driving image are calculated based on the calculated position change value. It may further include a shake data calculation unit to calculate.

In addition, the AR expression image generation unit may correct a position in the AR expression image of the AR guide object using the calculated shake data.

In addition, the display unit may further include an augmented reality image generated according to a combination of the second driving image at the second frame rate and the AR display image at the second frame rate.

In addition, a crop image generation unit configured to determine a crop region to cut a part of the second driving image by using the calculated shake data, and to generate a cropped image from the determined crop region of the second driving image. It can contain.

In addition, according to a combination of the third driving image at the second frame rate and the AR display image at the second frame rate, the third driving image is enlarged to the size of the second driving image. A display unit for displaying the generated augmented reality image may be further included.

On the other hand, the electronic device according to an embodiment of the present invention for achieving the above object, the image acquisition unit for acquiring the first driving image of the first frame rate photographed while the moving object, the obtained first driving image A driving image generation unit generating a second driving image at a second frame rate by processing the AR display image including the AR (Augmented Reality) guide object to be displayed on the second driving image by analyzing the second driving image. The AR display image generating unit generating the second frame rate, the image combining unit synchronizing and combining the generated second driving image and the AR displayed image, and the driving image of the second frame rate and the second frame rate. It may include a display unit for displaying the augmented reality image generated according to the combination of the AR display image.

On the other hand, a computer-readable recording medium according to an embodiment of the present invention for achieving the above object may be recorded a program for executing the above-described method.

In addition, code for executing the above-described method may be recorded in a program according to an embodiment of the present invention for achieving the above-described object.

According to various embodiments of the present invention described above, the frame rate of a real-time driving image is reduced to a frame rate within a minimum value that can be recognized as a video at a user's time and a processor performance maximum value, and the driving image of the changed frame rate is analyzed to guide AR By generating an AR expression image including an object and matching the frame rates of the driving image and the AR expression images, the AR guide object can be displayed at an accurate location in a real-time driving image containing the real world that the user is actually viewing. .

In addition, according to the present invention, by correcting the shaking of the driving image according to the shaking of the camera to provide a shaking-free driving image and at the same time correcting the display position of the AR guide object, real-time driving image containing the real world that the user is actually viewing. In the AR guide object can be displayed at the correct location.

Accordingly, the present invention provides an accurate augmented reality navigation service to the driver by displaying the AR object in the driving image at the same time while displaying the driving image containing the real world that the driver is actually viewing in the driver's view at the same time. can do.

Further, according to the present invention, by considering the performance of the processor in the range of change of the frame rate, it is possible to efficiently use the resources of the system.

1 is a block diagram showing an augmented reality image processing apparatus according to an embodiment of the present invention.
2 is a block diagram showing an augmented reality image processing apparatus according to an embodiment of the present invention in more detail.
3 is a block diagram specifically showing an AR-expressing image generating unit 13 according to an embodiment of the present invention.
4 is a view showing a conversion relationship between a 2D captured image and a virtual 3D space according to an embodiment of the present invention.
5 is a view showing the operation of the driving image generating unit according to an embodiment of the present invention.
6 is a view showing the operation of the AR expression image generation unit according to an embodiment of the present invention.
7 is a conceptual diagram for explaining a problem in a conventional augmented reality image processing method.
8 is a conceptual diagram illustrating an augmented reality image processing method according to an embodiment of the present invention.
9 is an exemplary view showing a driving image according to shaking.
10 is a conceptual diagram illustrating a process of extracting a feature point and calculating a position change value of the feature point according to an embodiment of the present invention.
11 to 12 are views showing a crop image generation process according to whether or not shaking occurs.
13 is a flowchart illustrating an augmented reality image processing method according to an embodiment of the present invention.
14 is a flowchart illustrating an augmented reality image processing method according to another embodiment of the present invention.
15 is a block diagram illustrating an electronic device according to an embodiment of the present invention.
16 is a diagram for describing a system network connected to an electronic device according to an embodiment of the present invention.
17 is a view showing a front UI collision prevention UI screen of an electronic device according to an embodiment of the present invention.
18 is a diagram illustrating an implementation form when an electronic device according to an embodiment of the present invention does not include a photographing unit.
19 is a diagram illustrating an implementation form when an electronic device according to an embodiment of the present invention includes a photographing unit.
20 is a view showing an implementation form using a head-up display (HUD) according to an embodiment of the present invention.

The following merely illustrates the principles of the present invention. Therefore, a person skilled in the art can implement various principles included in the concept and scope of the present invention and implement the principles of the present invention, although not explicitly described or illustrated in the present specification. In addition, all conditional terms and examples listed in this specification are intended to be understood in principle only for the purpose of understanding the concept of the present invention, and should be understood as not limited to the examples and states specifically listed in this way. do.

In addition, it should be understood that all detailed descriptions that list particular embodiments, as well as the principles, aspects and embodiments of the invention, are intended to include structural and functional equivalents of these matters. It should also be understood that these equivalents include all currently invented equivalents as well as equivalents to be developed in the future, ie, all devices invented to perform the same function regardless of structure.

Thus, for example, it should be understood that the block diagrams herein represent conceptual views of exemplary circuits embodying the principles of the invention. Similarly, all flow diagrams, state diagrams, pseudo-codes, etc. are understood to represent various processes performed by a computer or processor, whether or not the computer or processor is clearly depicted, which may be substantially represented on a computer-readable medium. Should be.

The functionality of the various elements shown in the figures, including a processor or functional block marked with a similar concept, may be provided by the use of dedicated hardware as well as hardware capable of executing software in connection with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared.

Also, the clear use of terms presented in terms of processors, controls, or similar concepts should not be interpreted exclusively by reference to hardware capable of executing software, and without limitation, digital signal processor (DSP) hardware, ROM for storing software. It should be understood to implicitly include (ROM), RAM and non-volatile memory. Other hardware for governors may also be included.

In the claims of the present specification, components expressed as means for performing the functions described in the detailed description include all types of software including, for example, combinations of circuit elements performing the above functions, firmware/micro code, and the like. It is intended to include all methods of performing a function, and is combined with appropriate circuitry for executing the software to perform the function. The present invention, as defined by these claims, is equivalent to any means capable of providing such functions are understood from this specification, since the functions provided by the various listed means are combined and combined with the manner required by the claims. It should be understood as.

The above objects, features, and advantages will become more apparent through the following detailed description in connection with the accompanying drawings, and accordingly, those skilled in the art to which the present invention pertains can easily implement the technical spirit of the present invention. There will be. In addition, in the description of the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing an augmented reality image processing apparatus according to an embodiment of the present invention. 2 is a block diagram showing an augmented reality image processing apparatus according to an embodiment of the present invention in more detail. 1 to 2, the augmented reality image processing apparatus 10 includes an image acquisition unit 11, a driving image generation unit 12, an AR expression image generation unit 13, an image combination unit 14, and shake data It may include a calculation unit 15, a crop image generation unit 16, a display unit 17.

The augmented reality image processing apparatus 10 lowers the frame rate of the driving image photographed while driving the vehicle to a frame rate within a minimum value that can be recognized as a video at a user's time and a processor performance maximum, and the changed frame By analyzing the driving image of the rate, an AR-expressing image including the AR guide object is generated, and the frame rates of the driving image and the AR-presenting images are matched and combined to generate an augmented reality image. Here, the driving image photographed while driving the vehicle may be referred to as a first driving image, and a frame rate of the first driving image may be referred to as a first frame rate. In addition, a frame rate within a minimum value that can be recognized as a video at a user's time and a maximum performance of the processor may be referred to as a second frame rate, and a driving image lowered to the second frame rate may be referred to as a second driving image. have. And, the frame rate unit may be defined as frame per second (fps), which means the number of frames per second.

In addition, the augmented reality image processing apparatus 10 may correct the shaking of the driving image according to the shaking of the photographing device installed in the vehicle, provide a shaking-free driving image, and at the same time correct the display position of the AR guide object.

Here, the augmented reality image processing apparatus 10 may be implemented using software, hardware, or a combination thereof. For example, according to a hardware implementation, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors (processors) ), controllers, micro-controllers, micro-processors, and other electrical units for performing other functions.

And, the moving object is an object that is movable and requires location measurement, and may be, for example, a person, a dog, a vehicle, a ship, a motorcycle, a bicycle, or a train. Hereinafter, for convenience of description, a case where the mobile body is implemented as a vehicle will be described in more detail with respect to each configuration module constituting the augmented reality image processing apparatus 10 as an example.

The image acquisition unit 11 may acquire, in real time, a first driving image at a first frame rate captured by a photographing device installed in the vehicle while the vehicle is driving. Here, the obtained first driving image may include a plurality of lanes distinguished along the lane, a road composed of a plurality of lanes, a plurality of vehicles driving the road, and other backgrounds.

Here, the lane may mean each of two lines forming a lane in which the vehicle is located. In addition, the lane (lane) is formed by a lane, such as the first lane, the second lane, .. N lane, etc., may mean a road on which the vehicle travels.

The driving image generation unit 12 may process the first driving image acquired by the image acquisition unit 11 to generate a second driving image at a second frame rate. Specifically, the driving image generation unit 120 may obtain a second frame rate within the maximum value of the processor and a maximum value that can be recognized as a moving image of the first driving image of the high frame rate acquired by the image acquisition unit 11 at a user's time. A second driving image lowered by can be generated. That is, the second frame rate may be smaller than the first frame rate.

For example, the first frame rate of the first driving image obtained by the image acquisition unit 11 may be 60 fps, and the driving image generation unit 12 processes the first driving image at 60 fps to determine the second frame rate. A second driving image may be generated. Here, the second frame rate is 15 fps or more, which is a minimum value that can be recognized as a video at the user's time, and the maximum value can be variably determined according to the performance of the processor. Specifically, the second frame rate may be increased when the processor processing the driving images of the AR guide objects is high-performance, and the second frame rate may be lowered when the processor is low-performance. In this way, system resources can be efficiently used by varying the second frame rate according to the performance of the processor.

Also, the second frame rate may be variably determined according to attribute information of an analysis target included in the second driving image. Specifically, the second driving image may include a plurality of objects (hereinafter, an analysis object) that require analysis to provide guidance to the vehicle driver. As an example, the analysis target may include various targets that require analysis to assist the vehicle driver, such as lanes, front vehicles, traffic lights, road signs, pedestrian crossings, pedestrians, and lanes included in the second driving image, and these elements This can affect image processing speed. Accordingly, the driving image generating unit 12 according to the present invention includes the number of analysis objects included in the second driving image, the image size of the analysis object included in the second driving image, and the type of analysis object included in the second driving image. (TYPE), the second frame rate may be variably determined according to attribute information of the analysis object including at least one of the data size of the analysis object included in the second driving image. For example, when a plurality of analysis objects are included in the second driving image, the frame rate may be lowered as the number of analysis objects increases in the second driving image in order to increase image processing speed.

Also, the second frame rate may be variably determined according to attribute information of the AR guide object. Specifically, the AR expression image generation unit 13, which will be described later, analyzes the second driving image to generate an AR guide object, determines a mapping location of the generated AR guide object, and generates an AR expression image based thereon. Here, the generation of the AR expression image is affected by the image processing speed for the generation of the AR guide object. Accordingly, the driving image generating unit 12 according to the present invention includes at least one of the type of the AR guide object, the image size of the AR guide object, the data size of the AR guide object, and the location where the AR guide object will be displayed. The second frame rate may be variably determined according to the attribute information of the AR guide object. For example, when the data size of the AR guide object is large, the second frame rate may be lowered as the data size increases to increase the image processing speed.

Meanwhile, the frame rate may be statistically determined in consideration of attribute information of a plurality of AR guide objects. For example, the frame rate may be determined according to the average image size of AR guide objects. In other words, in order to increase the image processing speed compared to the performance of the processor, the frame rate may become unfamiliar as the average size of the AR guide objects increases. As another example, the frame rate may be determined according to the average data size of AR guide objects. That is, in order to increase the image processing speed compared to the performance of the processor, as the average data size of the AR guide objects increases, the frame rate may become unfamiliar.

As described above, by determining the frame rate according to the attribute information of the analysis target included in the second driving image and the attribute information of the AR guide object, the augmented reality image processing apparatus 10 detects a matching error between the second driving image and the AR display image. The optimal frame rate that can be minimized can be determined.

Meanwhile, the present invention is not limited to the above embodiment, and the above-described frame rate may be determined by a bitrate, image size, and codec type of the second driving image. Here, the bitrate refers to the number of bits processed for a specific time unit (eg, seconds).

Meanwhile, the driving image generation unit 12 may generate a second driving image at the second frame rate through a process of selecting only a part of the first driving image at the first frame rate obtained by the image acquisition unit 11. . As an example, the first frame rate of the first driving image obtained by the image acquisition unit 11 may be 60 fps, and the driving image generation unit 12 only 15 frames from the first driving image composed of 60 frames per second By selecting, a 15 fps second driving image can be generated.

The AR expression image generation unit 13 analyzes the second driving image generated by the driving image generation unit 12 and displays an AR expression image including an AR (Augmented Reality) guide object to be displayed on the second driving image in the second frame. Can be created at a rate. Here, an example of the AR expression image may be a form in which at least one AR guide object providing driving-related guidance to the driver is disposed on a transparent background. In addition, the arrangement position of the AR guide object in the AR display image may vary according to the type of the AR guide object. In addition, the AR display image and the second driving image may have the same image size so that the AR display image matches the second driving image.

The AR expression image generation unit 13 will be described in more detail with reference to FIGS. 3 to 4.

3 is a block diagram specifically showing an AR-expressing image generator according to an embodiment of the present invention. Referring to FIG. 3, the AR expression image generation unit 13 includes a calibration unit 13-1, a 3D spatial generation unit 13-2, an AR guide object generation unit 13-3, and a mapping location determination unit 13 -4).

The calibration unit 13-1 may perform calibration for estimating camera parameters corresponding to the camera from the captured image taken by the camera. Here, the camera parameter may be a parameter constituting a camera matrix, which is information indicating a relationship that a photorealistic space has on a photo. Here, as shown in FIG. 4(a), the parameters may include extrinsic parameters 601 and intrinsic parameters 602. Here, fx, fy of the camera internal parameter 602 may be a focal length, cx, cy may be a principal point, and skew_c = tanα may be a skew coefficient. In addition, the camera external parameter 601 is the coordinate (X, Y, Z) of the 3D point on the world coordinate system (world coordinate system) 605, the coordinate of the 3D point on the camera coordinate system (Camera Coordinate System) 604 (Xc, Yc, Zc). Since the external parameters of the camera are not unique to the camera, it may vary depending on the location in which the camera is installed and in what direction, and may also depend on how the world coordinate system is defined.

That is, the calibration unit 13-1 according to the present invention may perform calibration to estimate camera parameters corresponding to the camera from the second driving image generated by the driving image generation unit 12.

The 3D space generation unit 13-2 may generate a virtual 3D space based on the captured image captured by the camera. Specifically, the 3D spatial generation unit 13-2 acquires depth information from the image captured by the camera based on the camera parameters estimated by the calibration unit 13-1, and acquires the acquired depth information and A virtual 3D space can be created based on an image. Specifically, referring to FIG. 4(b), the camera image is obtained by projecting points on a three-dimensional space of the world coordinate system 605 onto a two-dimensional image plane of the image coordinate system 603. Therefore, the 3D space generating unit 13-2 may perform a reverse process of the above-described operation based on the camera parameters to generate a virtual 3D (3-Dimensional) space of the world coordinate system 605 for the captured image of the camera. have.

That is, the 3D spatial generation unit 13-2 according to the present invention acquires depth information from the second driving image based on the camera parameters estimated by the calibration unit 13-1, and obtains the acquired depth information And a second driving image may generate a virtual 3D space.

The AR guide object generating unit 13-3 may generate an object for guidance in augmented reality, for example, a route guide object, a lane change guide object, a lane departure guide object, a pedestrian crossing object, a pedestrian guide object, and the like. . Here, the object may be implemented as a 3D object, a texture image or an art line.

To this end, the AR guide object generation unit 13-3 according to an embodiment of the present invention may include an image analysis module that analyzes frames of the second driving image generated in the driving image generation unit 12 in real time. have. The image analysis module detects an analysis object by analyzing frames of the second driving image in real time, determines what guidance is needed for the driver of the vehicle based on the detected analysis object, and augmented reality guidance object corresponding to the required guidance. Can be created. Here, the analysis target means an object that needs to be analyzed to provide guidance to the vehicle driver, for example, lanes, front vehicles, traffic lights, road signs, pedestrian crossings, pedestrians, lanes, etc. included in the second driving image It can include various objects that require analysis to assist the vehicle driver. For example, when a pedestrian crossing located in front of the driving of the vehicle is detected from the second driving image, the AR guide object generating unit 13-3 may generate a pedestrian crossing guide object. As another example, when the lanes located on the left and right sides of the vehicle are detected from the second driving image and it is determined that the vehicle is leaving the lane, the AR guide object generating unit 13-3 may generate a lane departure guide object. As another example, when the front vehicle located in front of the vehicle is detected from the second driving image and the distance between the vehicle and the front vehicle is determined to be less than a predetermined distance, the AR guide object generating unit 13-3 guides the front vehicle collision warning You can create objects.

That is, the AR guide object generating unit 13-3 according to an embodiment of the present invention does not use the frames of the first driving image acquired by the image obtaining unit 11 for analysis, and the driving image generating unit 12 The frames of the second driving image generated by selecting only a portion of the frames of the first driving image may be used for analysis.

On the other hand, the present invention is not limited to the above example, and the AR guide object generating unit 13-3 includes various data acquired while the vehicle is driving, for example, a second driving image captured while the vehicle is driving, and the current position of the vehicle. , By analyzing the driving status (driving speed, direction indication direction, fuel amount, etc.) obtained from the vehicle's ECU, and the setting status of the operation mode (whether the route guidance mode is set, whether the ADAS function is set, etc.) It is possible to determine whether it is necessary, and to generate an augmented reality guide object corresponding to the required guide.

The mapping location determining unit 13-4 may determine the mapping location of the object generated by the AR guide object generating unit 13-3 in the virtual 3D space generated by the 3D spatial generating unit 13-2.

That is, the mapping position determining unit 13-4 according to the present invention determines an appropriate mapping position in the virtual 3D space according to the type of the guide object generated by the AR guide object generating unit 13-3, and the determined mapping position You can create virtual 3D spatial data mapped to. For example, when a front vehicle collision prevention guidance is required, the mapping position determining unit 13-4 according to an embodiment of the present invention detects a pixel region of the front vehicle in a virtual 3D space, and detects the detected pixel region in the front vehicle. It may be determined as a mapping location of the AR guide object for collision avoidance guidance. Accordingly, the AR guide object may be superimposed and displayed on the front vehicle display area of the augmented reality image. As another example, when crosswalk guidance is required, the mapping location determining unit 13-4 according to an embodiment of the present invention detects a pixel region of a crosswalk in a virtual 3D space and guides the detected pixel region rather than crossing. It can be determined as the mapping location of the AR guide object for. Accordingly, the AR guide object may be superimposed and displayed on the crosswalk display area of the augmented reality image. As another example, when lane departure guidance is required, the mapping position determining unit 13-4 according to an embodiment of the present invention detects the pixel areas of the left and right lanes of the vehicle in the virtual 3D space and lanes the detected pixel areas It may be determined as a mapping location of the AR guide object for departure guidance. Accordingly, the AR guide object may be superimposed and displayed on the left and right lane areas of the vehicle of the augmented reality image.

Meanwhile, the AR expression image generation unit 13 according to an embodiment of the present invention may generate an AR expression image by converting a virtual 3D space to which an AR guide object is mapped into a 2D image using a camera matrix. In addition, the AR expression image generation unit 13 may generate the AR expression image at the same frame rate as the second driving image generated by the driving image generation unit 12. That is, the AR display image may be a 2D image and may be generated at the same second frame rate as the second driving image at the second frame rate.

On the other hand, in FIG. 3, it has been described as an example of determining the AR display image through 3D spatial data generation, AR guide object mapping, and 2D image generation using camera parameters, but is not limited thereto. According to another embodiment of the present invention, the step of generating an AR expression image including the AR guidance object at a second frame rate is determined by determining a position in the AR expression image of the AR guidance object by analyzing the second driving image. It may be configured as a step of generating an AR expression image in which the AR guide object is positioned at a position in the image. For example, when a front vehicle collision prevention guidance is required, according to another embodiment of the present invention, the second driving image generated by the driving image generator 12 is analyzed to detect a pixel area of the front vehicle, and the detected pixel area It may be determined as the display position of the AR guide object for the front vehicle collision prevention guidance. In addition, the AR display image generation unit 140 may generate an AR display image in which an AR guide object for preventing a front vehicle collision collision is located in the detected pixel area.

Meanwhile, the image combining unit 140 synchronizes the second driving image at the second frame rate generated by the driving image generator 12 and the AR displayed image at the second frame rate generated by the AR expressed image generator 13. Can be combined. Here, synchronization may refer to coincidence of a display time of a plurality of frames constituting a driving image and a display time of a plurality of frames constituting an AR display image. For example, when the frame rates of the second driving image generated by the driving image generating unit 12 and the AR displaying image generated by the AR displaying image generating unit 13 are both 15 fps, the image combining unit 140 is 1 second. Frame 1, frame 2, .. frame 15 constituting the second driving image and the display time of frame 1, frame 2, .. frame 15 constituting the AR display image can be combined by matching. . That is, the image combining unit 140 combines frame 1 of the second driving image and frame 1 of the AR displayed image based on 1 second, and combines frame N of the second driving image and frame N of the AR displayed image. It can be combined by matching the presentation time of the frames constituting the image.

According to the present invention, a frame rate of a real-time driving image is reduced to a frame rate within a minimum value that can be recognized as a video at a user's time and a processor performance maximum value, and the driving image of the changed frame rate is analyzed to include an AR guide object. By generating the AR display image and matching the display timings of the frames of the driving image and the frame rates of the AR display image, the AR guide object is displayed at the correct position in the real-time driving image containing the real world that the user is actually viewing. Can.

Hereinafter, an augmented reality image processing method according to an embodiment of the present invention will be described in detail with reference to the drawings.

5 is a view showing the operation of the driving image generating unit according to an embodiment of the present invention. Referring to FIG. 5, the driving image generator 12 may process the first driving image acquired by the image obtaining unit 11 to generate a second driving image at a second frame rate. In this case, the driving eternal life generation unit 12 sets a frame rate selection interval based on the frame rate conversion ratio between the first frame rate and the second frame rate, and removes the first frame rate according to the set frame rate selection interval. A second driving image at a second frame rate may be generated by selecting some frames from one driving image.

For example, when the first frame rate of the first driving image obtained by the image acquisition unit 11 is 4 fps, as shown in FIG. 5(a), the first driving image of 1 second is frame #1, frame #2, frame It may be composed of #3 and frame #4. In this situation, when the second frame rate is set to 2 fps as shown in FIG. 5(b), the driving image generation unit 12 sets the frame rate selection interval to 2 frames, and the frames acquired by the image acquisition unit 11 Frame #1 and frame #3 can be selected according to the selection interval among #1, frame #2, frame #3, and frame #4.

As another example, when the first frame rate of the first driving image obtained by the image acquisition unit 11 is set to 60 fps, the first driving image of 1 second may be composed of a total of 60 frames. In this situation, when the second frame rate is set to 15 fps, the driving image generation unit 12 sets the frame rate selection interval to 4 frames, and the set period among the total of 60 frames acquired by the image acquisition unit 11 According to the frame #1, #5, #9, #13, #17, #21, #25, #29, #33, #37, #41, #45, #49, #53, #57 can be selected have.

According to the present invention, in the process of changing the frame rate, by selecting a part of the frame at regular intervals and generating a second driving image, a phenomenon in which a specific part is cut off and displayed when the second driving image is displayed can be eliminated.

6 is a view showing the operation of the AR expression image generation unit according to an embodiment of the present invention. First, referring to FIG. 6, the AR guide object generating unit 13-3 according to an embodiment of the present invention travels without using frames of the first driving image acquired by the image obtaining unit 11 for analysis Frames of the second driving image generated by selecting only some of the frames of the first driving image may be used in the image generation unit 12 for analysis.

Specifically, referring to FIG. 6(a), the AR guide object generating unit 13-3 may analyze the frame #1 constituting the second driving image and detect a front vehicle and a 60 km speed limit sign. In this case, the AR guide object generating unit 13-2 determines that the driver of the vehicle needs to be guided by the front vehicle and the speed limit, and the AR guide object 1301 and the speed limit for guiding the front vehicle respectively to guide the vehicle. An AR guide object 1302 for guidance may be generated. And, referring to FIG. 6 (b), each of the AR guide object 1301 for front vehicle guidance and the AR guide object 1302 for speed limit guidance generated by the AR guide object generator 13-2 is mapped. According to the operation of the positioning unit 13-4, the AR display image frame #1 may be disposed at a predetermined position. Here, an example of the AR expression image may be a form in which at least one AR guide object providing driving-related guidance to the driver is disposed on a transparent background. Accordingly, the AR display image generation unit 13 may analyze the frame #1 of the second driving image to generate the AR display image frame #1.

On the other hand, referring to FIG. 6(c), in time frame #3 after frame #1, the distance between the vehicle and the vehicle in front is close, and the speed limit sign included in frame #1 disappears and a pedestrian crossing may be newly included. In this case, the AR guide object generating unit 13-3 may analyze the frame #3 constituting the second driving image to detect a vehicle in front and a pedestrian crossing with a distance closer to the previous one. In this case, the AR guide object generating unit 13-2 determines that the driver of the vehicle needs front vehicle guidance and crosswalk guidance, and the AR guide object 1304 and the crosswalk for guidance of the front vehicle to guide them respectively. An AR guide object 1303 for guidance may be generated. And, referring to FIG. 6 (d), each of the AR guide object 1304 for front vehicle guidance and the AR guide object 1303 for crosswalk guidance generated by the AR guide object generator 13-2 is mapped. According to the operation of the positioning unit 13-4, the AR display image frame #3 may be disposed at a predetermined position. In particular, since the position of the front vehicle in frame #3 is close to that of frame #1, the mapping position determining unit 13-4 displays the position of the AR guide object 1304 for guiding the front vehicle below the previous contrast. Can. Accordingly, the AR expression image generation unit 13 may generate the AR expression image frame #3 using frame #3 of the second driving image.

7 is a conceptual diagram for explaining a problem in a conventional augmented reality image processing method. According to the conventional augmented reality image processing method, since the frame per second of the real-time driving image that the driver sees in real time is different from the image processing frame per second using the image processing processor, the AR guide object is located at the correct position in the real-time driving image due to this mismatch. There was a problem that could not be expressed.

That is, according to the conventional method, when receiving a driving image of 60 fps as an example, the display position of the AR guide object is determined through image analysis on all 60 frames of the driving image on a per second basis, and the determined location is determined. A method of generating an AR expression image including the arranged AR guide object and combining the generated AR expression image and the driving image to display the screen is adopted.

However, according to this conventional method, in order to display the AR guide object on the driving image at the correct location, the analysis of the real-time driving image must be completed and the display position of the AR guide object must be determined before displaying the AR guide object. However, according to the conventional method, there is a problem in that analysis of all input frames is not completed before the display of the AR guide object according to the image processing performance of the processor. In this case, the display position of the AR guide object is not changed. The method of displaying as it is at the position determined in the previous frame was adopted.

For example, as shown in FIG. 7, when implemented to display a front vehicle collision prevention AR guide object in a pixel area occupied by a front vehicle in a real-time driving image, according to a conventional method, AR through analysis of frame #1 of the driving image When the analysis is completed before the display of the AR guide object, such as determining the display position of the guide object, the front vehicle collision prevention AR guide object 1101 is positioned and displayed in the front vehicle pixel area of frame #1. However, if the analysis for the input frame #2 is not completed before the display of the AR guide object after this, without changing the display position of the AR guide object 1102, the AR guide object 1102 remains at the position determined in the previous frame #1. Was marked.

Then, when the analysis is completed before the display of the AR guide object, such as the location where the AR guide object is displayed through the analysis of the input frame #3, the front vehicle pixel area of frame #3 prevents the front vehicle collision AR guide object 1103 ). However, if the analysis for the input frame #4 is not completed before the display of the AR guide object, the AR guide object 1104 remains at the position determined in the previous frame #3 without changing the display position of the AR guide object 1104. Was marked.

That is, according to the conventional augmented reality image processing method, there is a problem in that the AR guide object cannot be displayed at an accurate location in a real-time driving image containing the real world that the user is actually viewing. In particular, since augmented reality navigation is a combination of real-time driving images and AR guide objects that contain the real world so that it looks like objects existing in the original environment, it is very important that information is displayed at the correct location. It was difficult to achieve the purpose, and as a result, there was a problem that not only hindered the convenience and stability of the driver, but also dropped the driver's trust in navigation.

8 is a conceptual diagram illustrating an augmented reality image processing method according to an embodiment of the present invention. Referring to FIG. 8, the driving image generator 12 according to an embodiment of the present invention sets a frame rate selection interval based on a frame rate conversion ratio between a first frame rate and a second frame rate, and sets the set frame A second driving image at a second frame rate may be generated by selecting some frames from the first driving image at a first frame rate according to a rate selection interval. For example, as illustrated in FIG. 8, when the first frame rate of the first driving image acquired by the image acquisition unit 11 is 4 fps and the second frame rate is set to 2 fps, the driving image generation unit 12 may have a frame rate. The selection interval is set to 2 frames, and frame #1 and frame #3 among frame #1, frame #2, frame #3, and frame #4 acquired by the image acquisition unit 11 may be selected. Then, the driving image generator 12 may generate a second driving image composed of the selected frame #1 and frame #3.

In this case, the AR expression image generation unit 13 according to an embodiment of the present invention analyzes only the frames #1 and #3 of the second driving image generated by the driving image generation unit 12, and based on the analysis Accordingly, an AR expression image including an AR guide object to be displayed on the second driving image may be generated at a second frame rate.

For example, as shown in FIG. 8, when implemented to display an AR guidance object for preventing a front vehicle collision in a pixel area occupied by a front vehicle in a real-time driving image, according to the present invention, AR guidance is performed through analysis of frame #1 of the driving image The display position of the object may be determined, and the AR guide object 1201 in front of the vehicle collision prevention area 1201 may be displayed in the pixel area of the vehicle in front of frame #1. Subsequently, the same image processing may be performed on the input frame #3 to display the front vehicle collision prevention AR guide object 1203 in the pixel region of the front vehicle of frame #3.

However, since frame #2 and frame #4 are included in the first driving image but not in the second driving image, the AR expression image generating unit 13 according to the present invention is in frame #2 and frame #4. No separate image analysis is performed.

According to the present invention, the performance of the processor is generated by changing the first frame rate of the real-time driving image to the second frame rate, which is the frame rate within the maximum performance of the processor, and analyzing it to generate an AR-expressing image including the AR guide object. Regardless of the real-time driving image containing the real world that the user is actually viewing, it is possible to solve the problem that the AR guide object is not displayed at the correct location.

In addition, since the changed second frame rate is more than the minimum value that can be recognized as a video at the user's perspective, the driving object containing the real world that the driver is actually viewing is shown as a video, and at the same time, the AR object is accurately positioned in the driving video. By displaying, it is possible to provide an accurate augmented reality navigation service to the driver.

On the other hand, the augmented reality image processing apparatus 10 according to an embodiment of the present invention compensates for the shaking of the real-time driving image captured while driving the vehicle, provides a moving image without shaking, and simultaneously displays the AR guide object at the correct location. Can. To this end, the augmented reality image processing apparatus 10 according to an embodiment of the present invention may further include a shake data calculator 15 and a crop image generator 16.

Specifically, a driving image photographed while driving the vehicle may not output an image desired by the user due to the shaking of the photographing device. For example, when there is no shaking in the photographing device, the photographed driving image may be the same as in FIG. 9(A). However, when vertical shaking occurs in the photographing device in the upward direction, the photographed driving image may be as shown in FIG. 9(B), and when vertical shaking occurs in the photographing device in the downward direction, the photographed driving image is illustrated in FIG. 9. (C), and when the horizontal shaking occurs in the left direction on the photographing device, the photographed driving image may be as shown in FIG. 9(D), and when the horizontal shaking occurs in the right direction on the photographing device, photographing The rendered driving image may be as shown in FIG. 9(E).

In this way, the driving image photographed while the vehicle is running may not output the image desired by the user due to the shaking of the photographing device. In this case, in augmented reality navigation, there is a problem in that a screen desired by a user is not output, such as an AR guide object displayed in an incorrect location in a real-time driving image.

To solve this problem, the shake data calculator 15 according to an embodiment of the present invention may extract a feature point from the second driving image and calculate a position change value of the extracted feature point. Then, the shake data calculator 15 may calculate shake data indicating the shake size and direction of the second driving image based on the calculated position change value.

Specifically, the shake data calculator 15 may include a feature point extractor (not shown) that extracts a plurality of feature points from the second driving image. Here, the plurality of feature points extracted by the feature point extraction unit is a point in which the image brightness value changes rapidly in the second driving image, and may be an edge of pixel or a corner point. In addition, the shake data calculator 15 may further include a feature point change data calculator (not shown) that calculates a position change value of the extracted feature points. The feature point extraction unit and the feature point change data calculation unit will be described in more detail with reference to FIG. 10.

10 is a conceptual diagram illustrating a process of extracting a feature point and calculating a position change value of the feature point according to an embodiment of the present invention. Referring to FIG. 10(a), the feature point extraction unit extracts feature points from an image, and for example, an image brightness value such as an edge of pixel or a corner point in an image changes rapidly. The point to be may correspond to feature points (for example, points displayed on the vehicle boundary of FIG. 10(a)). Here, the feature point extraction unit includes algorithms for extracting feature points from an image, such as Harris method, Scale Invariant Feature Transform (SIFT) method, oriented fast and rotated brief (ORB) method, FAST (Features from Accelerated Segment Test) method, Lucas-Kanade Method or the like.

Then, referring to FIG. 10( b), the feature point change data calculator may calculate a position change value of the feature points by comparing the location of the feature points in each of the plurality of frames constituting the second driving image. For example, the feature point change data calculator may calculate a position change value of the feature points by comparing the locations of the feature points extracted in the previous frame and the feature points extracted in the subsequent frame. Here, the position change value may be embodied as a position vector value including the position change direction and the position change size of the feature points.

Meanwhile, the crop image generation unit 16 may determine a crop region to cut a portion of the second driving image, and generate a crop image cut from the determined second crop image. Here, the crop area may be determined based on the shake data calculated by the shake data calculator 15.

Meanwhile, the display unit 17 generates a third driving image in which the size of the cropped image is enlarged to the size of the second driving image, and is used to combine the third driving image at the second frame rate and the AR display image at the second frame rate. Accordingly, the generated augmented reality image may be displayed. The shake correction process will be described in more detail with reference to FIGS. 11 to 12.

11 to 12 are views showing a crop image generation process according to whether or not shaking occurs. Referring to FIGS. 11 to 12, the crop image generation unit 16 displays a crop region as shown in FIGS. 11(B) and 12(B) on the second driving images as shown in FIGS. 11(A) and 12(A). It can be set, and as shown in FIGS. 11(C) and 12(C), a third driving image in which the size of the crop image is enlarged to the size of the second driving image can be generated. That is, when the second driving image is used, the crop image generation unit 16 may set and use the crop region without using the entire image.

For example, after cropping only 1728 X 972 among the second driving images (FIGS. 11(A) and 12(A)) of 1920 X 1080 resolution, the third driving images (FIGS. 11(C) and 12(C) are enlarged. )). Here, the crop area may be determined based on the shake data calculated by the shake data calculator 15.

For example, as shown in FIG. 11(B), when the shaking data calculated by the shaking data calculating unit 15 is 0 because no shaking occurs, the crop image generating unit 16 has a predetermined size centering on the center of the image. The crop area 16-1 can be set.

As another example, as shown in FIG. 12(B), when the shake data calculated by the shake data calculator 15 is 100 pixels at the top, the crop image generator 160 is 100 pixels down from the center of the image. It is possible to set the crop region 16-2 of a predetermined size with the moved point as the center point.

That is, the crop region may be set in the opposite direction of the shake data obtained through feature point extraction (ie, the opposite direction of the camera shake direction).

On the other hand, the cropped image generating unit 16 changes the cropping area in the opposite direction of the shake data, crops by 1728 X 972, and enlarges it again to obtain the third driving image (FIGS. 11(C) and 12(C)). Can be created.

According to the present invention, even if the camera is shaken and an accurate image is not captured, it can be shown to the user as an image that does not seem to shake.

Meanwhile, the display unit 17 may display the augmented reality image generated according to the combination of the third driving image at the second frame rate and the AR display image at the second frame rate. According to the present invention, a third driving image is generated based on a cropped image that compensates for the effects of shaking, and the third driving image and the AR display image are synchronized and combined to solve a matching error between the images to solve the AR guide object Can be displayed at the correct location.

However, this is only an embodiment of the present invention, and according to another embodiment of the present invention, the augmented reality image processing apparatus 10 may minimize image matching errors by performing image stabilization by a stabilizer method through hardware. . As an example, the augmented reality image processing apparatus 10 may further include a 3-axis sensor, and may control a camera position to be moved in real time based on the 3-axis sensor to correct image shake to minimize matching errors.

On the other hand, the above example is only one embodiment of the present invention, according to another embodiment of the present invention, by using the shake data calculated by the shake data calculation unit 15 by correcting the position in the AR display image of the AR guide object , In an augmented reality image, an AR guide object may be displayed at an accurate location. Specifically, the AR display image generation unit 13 may correct the position in the AR display image of the AR guide object using the shake data calculated by the shake data calculation unit 15. For example, according to the shaking data calculated by the shaking data calculating unit 15, the AR display image generating unit 13 does not reflect the shaking data in the state where the frame of the second driving image moves 100 pixels to the right of the previous frame. If the location of the guide object in the AR display image is determined, the AR guide object in the augmented reality image may not be displayed at the correct location due to the influence of shaking. Accordingly, the AR expression image generating unit 13 of the present invention performs correction to move the position of the AR guide object 100 pixels to the right in the AR expression image by reflecting the shaking data calculated by the shaking data calculation unit 15. can do. In this case, the AR guide object in the augmented reality image may be displayed at an accurate location by reflecting the influence of the shaking.

That is, the display unit 17 displays the augmented reality image generated according to the combination of the second driving image at the second frame rate and the AR displayed image at the second frame rate, and is displayed on the second driving image due to the shaking of the photographing device. Even if shaking occurs, by using the shaking data calculated by the shaking data calculating unit 15 to correct the position of the AR guide object in the AR display image, the AR guidance object in the augmented reality image reflects the effect of the shaking, so that Can be displayed.

Hereinafter, an augmented reality image processing method according to an embodiment of the present invention will be described in more detail with reference to FIGS. 13 to 14.

13 is a flowchart illustrating an augmented reality image processing method according to an embodiment of the present invention. Referring to FIG. 13, first, a first driving image of a first frame rate photographed while driving the mobile body may be obtained (S110 ).

Then, a second driving image at a second frame rate may be generated by processing the obtained first driving image (S120). Here, the second frame rate may be smaller than the first frame rate. For example, the second frame rate is 15 fps or more, which is a minimum value that can be recognized as a video at a user's time, and the maximum value can be variably determined according to the performance of the processor. Specifically, the second frame rate may be increased when the processor processing the driving images of the AR guide objects is high-performance, and the second frame rate may be lowered when the processor is low-performance. In this way, system resources can be efficiently used by varying the second frame rate according to the performance of the processor. In this way, by limiting the frame according to the performance of the processor, it is possible to efficiently use the resources of the system.

Also, the second frame rate may be variably determined according to attribute information of the AR guide object. Also, the second frame rate may be variably determined according to attribute information of an analysis target included in the second driving image.

On the other hand, according to the augmented reality image processing method, the second driving image may be analyzed to generate an AR expressed image including an Augmented Reality (AR) guide object to be displayed on the second driving image at the second frame rate (S130) ). Specifically, generating an AR expression image including the AR guidance object at a second frame rate (S130) includes analyzing a second driving image to determine a location within the AR expression image of the AR guidance object and a determined location within the image. And generating an AR expression image in which the AR guide object is positioned. For example, the AR expression image may be in the form of at least one AR guidance object that provides driving guidance to the driver on a transparent background. In addition, the arrangement position of the AR guide object in the AR display image may vary according to the type of the AR guide object. In addition, the AR display image and the second driving image may have the same image size so that the AR display image matches the second driving image.

Meanwhile, according to the augmented reality image processing method, the generated second driving image and the AR display image may be synchronized and combined (S140). Here, synchronization may refer to coincidence of a display time of a plurality of frames constituting a driving image and a display time of a plurality of frames constituting an AR display image. Accordingly, when the second driving image is overlapped with the transparent area in which the AR guide object is not disposed in the AR display image, only the second driving image is displayed in the overlapping region, and the area and the AR region in which the AR guide object is placed in the AR display image are displayed. 2 When the driving image overlaps, the AR guide object may be displayed in the overlapping region.

According to the present invention, a frame rate of a real-time driving image is reduced to a frame rate within a minimum value that can be recognized as a video at a user's time and a processor performance maximum value, and the driving image of the changed frame rate is analyzed to include an AR guide object. By generating the AR display image and matching the frame rates of the driving image and the AR display images, it is possible to display the AR guide object at the correct location in the real-time driving image containing the real world that the user is actually viewing.

14 is a flowchart illustrating an augmented reality image processing method according to another embodiment of the present invention. Referring to FIG. 14, first, a first driving image at a first frame rate photographed while driving the mobile body may be obtained (S210 ). A second driving image at a second frame rate may be generated by processing the acquired first driving image (S220).

Then, a crop region to cut a portion of the second driving image may be determined (S230 ), and a cropped image cut from the determined crop region may be generated (S240 ). Here, the crop area may be determined based on the shake data calculated by the shake data calculator 15. That is, the crop region may be determined in the opposite direction of the shake data obtained through feature point extraction (ie, the opposite direction of the camera shake direction).

On the other hand, according to the augmented reality image processing method of the present invention, by expanding the size of the cropped image to the size of the second driving image to generate a third driving image at a second frame rate (S250), and analyze the third driving image An AR expression image including an AR (Augmented Reality) guide object to be displayed on the third driving image may be generated at the second frame rate (S260).

Then, the third driving image at the second frame rate and the AR display image at the second frame rate may be synchronized and combined (S270). According to the present invention, a third driving image is generated based on a cropped image that compensates for the effects of shaking, and the third driving image and the AR display image are synchronized and combined to solve the matching error between the images to solve the AR guide object Can be displayed at the correct location.

Through the augmented reality image processing method described above, the augmented reality image processing apparatus 10 may provide an augmented reality navigation to a driver by allowing the augmented reality object to be displayed at an accurate location even in a real-time image. In addition, the augmented reality image processing apparatus 10 may provide an augmented reality navigation to a driver by allowing the augmented reality object to be displayed at an accurate location in a shaking real-time image.

Meanwhile, the augmented reality image processing apparatus 10 may be implemented as a module of an electronic device that outputs various guide information for assisting a driver's driving or a system for autonomous driving, and performs a route guidance function. This will be described in more detail with reference to FIG. 15.

15 is a block diagram illustrating an electronic device according to an embodiment of the present invention. 15, the electronic device 100 includes a storage unit 110, an input unit 120, an output unit 130, an augmented reality providing unit 160, a control unit 170, a communication unit 180, and a sensing unit ( 190), all or part of the power supply unit 195.

Here, the electronic device 100 is a smart phone, a tablet computer, a notebook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a smart glass, a project glass, navigation that can provide driving guidance to the driver of the vehicle. (navigation), can be implemented in various devices, such as a car dash cam or car video recorder, which is a vehicle image photographing device, and may be provided in a vehicle.

Driving guidance includes route guidance, lane departure guidance, lane maintenance guidance, front vehicle departure guidance, traffic light change guidance, front vehicle collision prevention guidance, lane change guidance, lane guidance, curve guidance, speed limit guidance, vehicle width limit guidance, vehicle weight It may include various guides for assisting the driving of the vehicle driver, such as the guide limit and the vehicle height limit guide.

Here, the route guidance is augmented reality route guidance, 2D (2-Dimensional) or 3D (3-Dimensional), which performs route guidance by combining various information such as a user's location and direction with an image photographing the front of a running vehicle. It may include 2D (2-Dimensional) or 3D (3-Dimensional) route guidance by combining various information such as the user's location and direction with the map data of.

In addition, the route guidance may include an aerial map route guidance that combines various information such as a user's location and direction in the aerial map data to perform route guidance. Here, the route guide may be interpreted as a concept including a route guide when the user walks or jumps, as well as when the user rides and drives the vehicle.

In addition, the lane departure guidance may be to guide whether the vehicle being driven has left the lane.

In addition, the lane maintenance guidance may be to guide the vehicle to return to the original driving lane.

In addition, the front vehicle departure guide may be to guide the departure of a vehicle located in front of the vehicle being stopped.

In addition, the traffic light change guide may be to guide whether the signal of the traffic light located in front of the vehicle being stopped is changed. For example, when the red traffic light indicating the stop signal is turned on and the blue traffic light indicating the start signal is changed, it may be guided.

In addition, the front vehicle collision prevention guide may be to guide a vehicle in front of a vehicle that is stopped or running when the distance to the vehicle is within a predetermined distance to prevent collision with the vehicle in front.

In addition, the lane change guide may be to guide the change from the lane where the vehicle is located to another lane to guide the route to the destination.

In addition, the lane guidance may be to guide the lane in which the current vehicle is located.

In addition, the curve guide may be a guide that the road on which the vehicle will travel after a predetermined time is a curve.

In addition, the speed limit guide may be to guide the speed limit of the lane in which the current vehicle is located or the lane to enter.

In addition, the vehicle width limitation guide may be to guide the vehicle width limitation of the lane in which the current vehicle is located or the lane to enter.

In addition, the vehicle weight limit guide may be to guide the vehicle weight limit of the lane in which the current vehicle is located or the lane to enter.

In addition, the vehicle height restriction guide may be to guide the vehicle height restriction of the lane in which the current vehicle is located or the lane to enter.

Such a driving-related image, such as a front image of a vehicle that enables providing various guidance, may be captured by a camera mounted on a vehicle or a camera of a smart phone. Here, the camera may be a camera that is integrally formed with the electronic device 100 mounted on the vehicle and photographs the front of the vehicle.

As another example, the camera may be a camera mounted on a vehicle separately from the electronic device 100 to photograph the front of the vehicle. In this case, the camera may be a separate vehicle image capturing device mounted toward the front of the vehicle, and the electronic device 100 may receive a captured image through wired/wireless communication with the separately mounted vehicle image capturing device, or a vehicle image When the storage medium for storing the captured image of the photographing device is inserted into the electronic device 100, the electronic device 100 may receive the photographed image.

Hereinafter, the electronic device 100 according to an exemplary embodiment of the present invention will be described in more detail based on the above-described contents.

The storage unit 110 functions to store various data and applications required for the operation of the electronic device 100. In particular, the storage unit 110 may store data necessary for the operation of the electronic device 100, for example, an OS, a route search application, map data, and the like. In addition, the storage unit 110 may store data generated by the operation of the electronic device 100, for example, searched route data, received images, and the like.

The storage unit 110 includes random access memory (RAM), flash memory, read only memory (ROM), erasable programmable ROM (EPROM), electronically erasable and programmable ROM (EEPROM), registers, hard disk, removable disk, memory It can be implemented as a built-in storage such as a card, a Universal Subscriber Identity Module (USIM), and a removable storage such as a USB memory.

The input unit 120 functions to convert a physical input from the outside of the electronic device 100 into a specific electrical signal. Here, the input unit 120 may include all or part of the user input unit 121 and the microphone unit 123.

The user input unit 121 may receive a user input such as a touch or push operation. Here, the user input unit 121 may be implemented using at least one of various types of buttons, a touch sensor receiving a touch input, and a proximity sensor receiving an approaching motion.

The microphone unit 123 may receive a user's voice and sound generated inside and outside the vehicle.

The output unit 130 is a device that outputs data of the electronic device 100 to a user through video and/or audio. Here, the output unit 130 may include all or part of the display unit 131 and the audio output unit 133. That is, the output unit 130 may serve as the display unit 14 of the augmented reality image processing apparatus 10.

The display unit 131 is a device that outputs data that can be visually recognized to a user. The display unit 131 may be implemented as a display unit provided on the front surface of the housing of the electronic device 100. In addition, the display unit 131 is integrally formed with the electronic device 100 to output visual recognition data, and is installed separately from the electronic device 100, such as a head up display (HUD), to output visual recognition data. You may.

The audio output unit 133 is a device that outputs data that the electronic device 100 can perceive acoustically. The audio output unit 133 may be implemented as a speaker that expresses sound of data to be informed to a user of the electronic device 100. That is, the audio output unit 133 may output a predetermined warning sound or driving guidance sound.

The augmented reality providing unit 160 may perform the functions of the augmented reality image processing apparatus 10 described above. Specifically, the augmented reality providing unit 160 provides an AR guide object representing additional information (for example, a point of interest (POI)) of a driving image containing the real world that the user is actually viewing, and the risk of collision in front of the vehicle. The AR guide object for guidance, the AR guide object for indicating the distance between vehicles, the AR guide object for guiding the curve, and various AR guide objects for helping the driver to safely drive) may be visually superimposed.

Meanwhile, the communication unit 180 may be provided for the electronic device 100 to communicate with other devices. The communication unit 180 may include all or part of the location data unit 181, the wireless Internet unit 183, the broadcast transmission/reception unit 185, the mobile communication unit 186, the local area communication unit 187, and the wired communication unit 189. Can.

The location data unit 181 is a device that acquires location data through a Global Navigation Satellite System (GNSS). GNSS means a navigation system that can calculate the location of a receiving terminal using a radio wave signal received from a satellite. Specific examples of GNSS include Global Positioning System (GPS), Galileo, Global Orbiting Navigational Satellite System (GLONASS), COMPASS, Indian Regional Navigational Satellite System (IRNSS), and Quasi-Zenith Satellite System (QZSS) depending on the operating entity. Can. The location data unit 181 of the electronic device 100 according to an embodiment of the present invention may obtain location data by receiving a GNSS signal serviced in an area where the system 100 is used. Alternatively, the location data unit 181 may acquire location data through communication with a base station or an AP (Access Point) in addition to the GNSS.

The wireless Internet unit 183 is a device that connects to the wireless Internet to acquire or transmit data. The wireless Internet unit 183 uses various communication protocols defined to perform wireless data transmission and reception of wireless LAN (WLAN), wireless broadband (Wibro), world interoperability for microwave access (Wimax), and high speed downlink packet access (HSDPA). You can connect to the Internet network.

The broadcast transmission/reception unit 185 is a device for transmitting and receiving broadcast signals through various broadcast systems. Broadcasting systems that can be transmitted and received through the broadcast transmitting and receiving unit 185 include Digital Multimedia Broadcasting Terrestrial (DMBT), Digital Multimedia Broadcasting Satellite (DMBS), Media Forward Link Only (FLOFLO), Digital Video Broadcast Handheld (DVBH), and Integrated SDSD (Integrated) Services Digital Broadcast Terrestrial). The broadcast signal transmitted/received through the broadcast transmitting/receiving unit 185 may include traffic data, living data, and the like.

The mobile communication unit 186 may connect to a mobile communication network according to various mobile communication standards such as 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE) to communicate with voice and data.

The short-range communication unit 187 is a device for short-range communication. The short-range communication unit 187, as described above, Bluetooth (Bluetooth), Radio Frequency Identification (RFID), infrared communication (IrDA, Infrared Data Association), UWB (Ultra WideBand), ZigBee, NFC (Near Field Communication), Wi It can communicate through -Fi (Wireless-Fidelity).

The wired communication unit 189 is an interface device that can connect the electronic device 100 to other devices by wire. The wired communication unit 189 may be a USB module that can communicate through a USB port.

The communication unit 180 includes at least one of a location data unit 181, a wireless Internet unit 183, a broadcast transmission/reception unit 185, a mobile communication unit 186, a local area communication unit 187, and a wired communication unit 189. Use to communicate with other devices.

For example, when the electronic device 100 does not include a camera function, at least one of a short-range communication unit 187 and a wired communication unit 189 may be used to record an image captured by a vehicle image capturing device such as a Car dash cam or a Car video recorder. You can use it to receive.

As another example, when communicating with a plurality of devices, one may communicate with the local area communication unit 187 and the other may communicate with the wired communication unit 119.

The sensing unit 190 is a device capable of detecting the current state of the system 100. The sensing unit 190 may include all or part of the motion sensing unit 191 and the light sensing unit 193.

The motion sensing unit 191 may detect motion in the 3D space of the electronic device 100. The motion sensing unit 191 may include a 3-axis geomagnetic sensor and a 3-axis acceleration sensor. The motion data obtained through the motion sensing unit 191 may be combined with the position data obtained through the location data unit 181 to more accurately calculate the trajectory of the vehicle to which the electronic device 100 is attached.

The optical sensing unit 193 is a device for measuring the ambient illumination of the system 100. By using the illuminance data acquired through the light sensing unit 193, the brightness of the display unit 131 may be changed to correspond to the ambient brightness.

The power supply unit 195 is a device that supplies power required for the operation of the electronic device 100 or other devices connected to the electronic device 100. The power supply unit 195 may be a device that receives power from an external power source such as a battery or a vehicle built in the electronic device 100. In addition, the power supply unit 195 may be implemented as a wired communication module 119 or a wirelessly supplied device depending on the form of receiving power.

The control unit 170 controls the overall operation of the electronic device 100. Specifically, the control unit 170 includes all of the storage unit 110, the input unit 120, the output unit 130, the augmented reality providing unit 160, the communication unit 180, the sensing unit 190, and the power supply unit 195, or Some can be controlled.

In particular, the control unit 170 may control the display unit 131 to display the augmented reality image generated by the augmented reality providing unit 160. Specifically, the augmented reality providing unit 160 acquires a first driving image at a first frame rate photographed while the mobile body is driving, and processes the obtained first driving image to generate a second driving image at a second frame rate. Then, the second driving image is analyzed to generate an AR display image including an AR (Augmented Reality) guide object to be displayed on the second driving image at a second frame rate, and the generated second driving image and the AR display image are synchronized. To combine to generate an augmented reality image. In this case, the control unit 170 may control the display unit 131 to display the augmented reality image generated by the augmented reality providing unit 160.

16 is a diagram for describing a system network connected to an electronic device according to an embodiment of the present invention. Referring to FIG. 16, the electronic device 100 according to an embodiment of the present invention may be implemented as various devices provided in a vehicle, such as a navigation device, an image capturing device for a vehicle, a smart phone, or other augmented reality interface providing device for a vehicle, and the like. , It can be connected to various communication networks and other electronic devices (61 to 64).

In addition, the electronic device 100 may calculate the current location and the current time zone by interlocking the GPS module according to the radio wave signal received from the satellite 20.

Each satellite 20 may transmit L band frequencies with different frequency bands. The electronic device 100 may calculate the current position based on the time taken for the L-band frequency transmitted from each satellite 20 to reach the electronic device 100.

Meanwhile, the electronic device 100 may wirelessly access the network 30 through the control unit 180 through the control station 40, ACR, base station 50, RAS, and access point (AP). When the electronic device 100 is connected to the network 30, data can be exchanged by indirectly connecting to other electronic devices 61 and 62 connected to the network 30.

Meanwhile, the electronic device 100 may indirectly connect to the network 30 through another device 63 having a communication function. For example, when the electronic device 100 is not provided with a module capable of accessing the network 30, it may communicate with another device 63 having a communication function through a short-range communication module.

17 is a diagram illustrating a vehicle collision prevention guide screen of an electronic device according to an embodiment of the present invention. Referring to FIG. 17, the electronic device 100 may generate a guide object indicating a vehicle collision risk and output the generated guide object 1001 through augmented reality.

Here, the guide object 1001 may be an object that guides that a user needs attention. That is, the vehicle collision prevention guide may be to guide that the vehicle is in danger of collision with the vehicle in front because the vehicle is narrowed within a predetermined distance. In this embodiment, the guide object 1001 is implemented as a texture image and can be expressed through augmented reality. Accordingly, the driver can easily recognize the road on which the host vehicle is driving.

Also, the electronic device 100 may output the guide object 1001 through voice.

18 is a diagram illustrating an implementation form when an electronic device according to an embodiment of the present invention does not include a photographing unit. Referring to FIG. 18, a vehicle image photographing device 200 provided separately from the electronic device 100 may configure a system according to an embodiment of the present invention using a wired/wireless communication method.

The electronic device 100 may include a display unit 131 provided on the front surface of the housing 191, a user input unit 121, and a microphone 123.

The vehicle image capturing apparatus 200 may include a camera 222, a microphone 224, and an attachment portion 281.

19 is a diagram illustrating an implementation form when an electronic device according to an embodiment of the present invention includes a photographing unit. Referring to FIG. 19, when the electronic device 100 includes the photographing unit 150, the user photographs the front of the vehicle by the photographing unit 150 of the electronic device 100 and displays the electronic device 100. It may be a device that allows the user to recognize the part. Accordingly, a system according to an embodiment of the present invention can be implemented.

20 is a view showing an implementation form using a head-up display (HUD) according to an embodiment of the present invention. Referring to FIG. 20, the HUD may display an augmented reality guide screen on a head-up display through wired/wireless communication with other devices.

For example, augmented reality may be provided through a HUD using a windshield of a vehicle or an image overlay using a separate image output device, and the augmented reality providing unit 160 may provide a real image or an interface image overlaid on the glass. You can create Through this, an augmented reality navigation or a vehicle infotainment system may be implemented.

On the other hand, according to one embodiment of the present invention, the augmented reality image processing apparatus 10 receives the first driving image at the first frame rate and generates a second driving image at the second frame rate as an example. , This is only an embodiment of the present invention and is not limited thereto. According to another embodiment of the present invention, a second driving image at a second frame rate may be generated from a first driving image at a first frame rate through an external separate module, and the augmented reality image processing apparatus 10 may be configured as an external module. It may be implemented in the form of receiving a second driving image at a second frame rate from.

On the other hand, terms such as "first", "second", "third" and "fourth" in the specification and claims, if any, are used to distinguish between similar elements, but not necessarily Used to describe a specific sequence or order of occurrence. It will be understood that the terms so used are compatible under appropriate circumstances so that the embodiments of the invention described herein can operate in sequences other than those shown or described herein, for example. Likewise, where the method is described as including a series of steps, the order of such steps presented herein is not necessarily the order in which such steps may be performed, and any described steps may be omitted and/or here Any other steps not described may be added to the method.

In addition, the terms "left", "right", "front", "back", "upper", "bottom", "above", "below", etc. in the specification and claims are used for explanation, It is not necessarily to describe the constant relative position. It will be understood that the terms so used are compatible under appropriate circumstances so that the embodiments of the invention described herein may operate in other directions than those illustrated or described herein, for example. The term "connected" as used herein is defined as being connected directly or indirectly in an electrical or non-electrical manner. Objects described herein as “adjacent” to one another may be physically in contact with one another, in close proximity to one another, or in the same general scope or area as appropriate for the context in which the phrases are used. The presence of the phrase "in one embodiment" here does not necessarily mean the same embodiment.

Also, in the specification and claims, reference to various variations of these expressions such as'connected','connected','concluded','concluded','coupled','combined', etc. is directly with other components. It is used in a sense that includes connecting or indirectly connecting through other components.

In addition, the suffixes "module" and "part" for the components used in the present specification are given or mixed only considering the ease of writing the specification, and do not have a meaning or a role distinguished from each other in itself.

Also, the terms used in the present specification are for describing the embodiments and are not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein,'comprise' and/or'comprising' refers to the elements, steps, operations and/or elements mentioned above, the presence of one or more other components, steps, operations and/or elements. Or do not exclude additions.

So far, the present invention has been focused on the preferred embodiments. All of the embodiments and conditional examples disclosed through this specification have been described with the intention of helping those skilled in the art to understand the principles and concepts of the present invention. It will be understood that this may be implemented in a modified form without departing from the essential characteristics of the present invention.

Therefore, the disclosed embodiments should be considered in terms of explanation, not limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be interpreted as being included in the present invention.

Meanwhile, the augmented reality image processing method according to various embodiments of the present invention described above may be implemented as a program and provided to a server or devices. Accordingly, each device can download the program by accessing a server or device in which the program is stored.

Meanwhile, the speed limit guide method according to various embodiments of the present invention described above may be implemented as a program and provided to a server or devices. Accordingly, each device can download the program by accessing a server or device in which the program is stored.

In addition, the control method according to various embodiments of the present invention described above may be implemented as a program and stored in various non-transitory computer readable media and provided. The non-transitory readable medium means a medium that stores data semi-permanently and that can be read by a device, rather than a medium that stores data for a short time, such as registers, caches, and memory. Specifically, the various applications or programs described above may be stored and provided in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. In addition, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical idea or prospect of the present invention.

10: augmented reality image processing apparatus 11: image acquisition unit
12: driving image generation unit 13: AR display image generation unit
14: image combining unit 15: shake data calculation unit
16: crop image generation unit 17: display unit
100: electronic device 110: storage unit
120: input unit 130: output unit
160: augmented reality providing unit 170: control unit
180: communication unit 190: sensing unit
195: power supply

Claims (23)

In the augmented reality image processing method using a processor,
Obtaining a first driving image at a first frame rate photographed while the mobile body is driving;
Generating a second driving image at a second frame rate by processing the acquired first driving image;
Analyzing the second driving image and generating an AR expression image including an Augmented Reality (AR) guide object to be displayed on the second driving image at the second frame rate;
Including; synchronizing and combining the generated second driving image and the AR display image;
The second frame rate is less than the first frame rate, augmented reality image processing method. According to claim 1,
The second frame rate is augmented reality image processing method characterized in that it is variably determined according to the performance of the processor. According to claim 1,
The second frame rate is augmented reality image processing method characterized in that it is variably determined according to the attribute information of the AR guide object. According to claim 1,
The second frame rate is statistically determined by considering attribute information of an analysis target included in the second driving image. According to claim 1,
Generating an AR expression image including the AR guide object at the second frame rate,
Determining a position in the AR display image of the AR guide object by analyzing the second driving image; And
And generating the AR-expressed image in which the AR guide object is positioned in the determined location within the image. According to claim 1,
Extracting a feature point from the second driving image and calculating a position change value of the extracted feature point; And
And calculating shake data indicating a shake size and a shake direction of the second driving image based on the calculated position change value. The method of claim 6,
And correcting a position of the AR guide object in the AR-expressed image using the calculated shake data. The method of claim 7,
And displaying the augmented reality image generated according to the combination of the second driving image at the second frame rate and the AR display image at the second frame rate. The method of claim 6,
Determining a crop region to cut a part of the second driving image using the calculated shake data; And
And generating a cropped image cropped from the determined cropping region of the second driving image. The method of claim 9,
Generating a third driving image in which the size of the cropped image is enlarged to the size of the second driving image; And
And displaying the augmented reality image generated according to the combination of the third driving image at the second frame rate and the AR display image at the second frame rate. In the augmented reality image processing apparatus,
An image acquiring unit acquiring a first driving image at a first frame rate photographed while the mobile body is driving;
A driving image generator configured to process the acquired first driving image to generate a second driving image at a second frame rate;
An AR expression image generation unit analyzing the second driving image and generating an AR expression image including an Augmented Reality (AR) guide object to be displayed on the second driving image at the second frame rate; And
Includes; an image combining unit for synchronizing and combining the generated second driving image and the AR display image;
The second frame rate is augmented reality image processing apparatus, characterized in that less than the first frame rate. The method of claim 11,
The second frame rate is augmented reality image processing apparatus characterized in that it is variably determined according to the performance of the processor. The method of claim 11,
The second frame rate is augmented reality image processing apparatus characterized in that it is variably determined according to the attribute information of the AR guide object. The method of claim 11,
The second frame rate is augmented reality image processing apparatus characterized in that it is variably determined according to the attribute information of the analysis target included in the second driving image. The method of claim 11,
The AR expression image generation unit,
Augmented reality image characterized by analyzing the second driving image to determine the location within the AR display image of the AR guide object, and generating the AR display image in which the AR guide object is positioned in the determined image location. Processing unit. The method of claim 11,
Extracting a feature point from the second driving image, calculating a position change value of the extracted feature point, and calculating shake data indicating a shake size and a shake direction of the second driving image based on the calculated position change value An augmented reality image processing apparatus further comprising a shake data calculation unit. The method of claim 16,
The AR expression image generation unit,
Augmented reality image processing apparatus characterized in that for correcting the position in the AR display image of the AR guide object using the calculated shake data. The method of claim 17,
And a display unit configured to display an augmented reality image generated according to a combination of the second driving image at the second frame rate and the AR display image at the second frame rate. The method of claim 16,
A crop image generator configured to determine a crop region to cut a part of the second driving image using the calculated shake data, and generate a cropped image from the determined crop region of the second driving image; Augmented reality image processing apparatus comprising a. The method of claim 19,
A third driving image in which the size of the cropped image is enlarged to the size of the second driving image is generated, and generated according to a combination of the third driving image at the second frame rate and the AR display image at the second frame rate. And an augmented reality image display unit. In electronic devices,
An image acquiring unit acquiring a first driving image at a first frame rate photographed while the mobile body is driving;
A driving image generator configured to process the acquired first driving image to generate a second driving image at a second frame rate;
An AR expression image generation unit analyzing the second driving image and generating an AR expression image including an Augmented Reality (AR) guide object to be displayed on the second driving image at the second frame rate;
An image combining unit synchronizing and combining the generated second driving image and the AR display image; And
And a display unit configured to display an augmented reality image generated according to a combination of the driving image at the second frame rate and the AR display image at the second frame rate. A computer-readable recording medium in which a program for executing the method according to any one of claims 1 to 10 is recorded. A program stored on a computer-readable recording medium and executing the method according to any one of claims 1 to 10.

Images