JP2020166741A - Image processing device for ar, image processing system, and program - Google Patents

Abstract

To highly accurately display an object in AR (Augmented Reality).SOLUTION: An image processing device includes a mark image input part to which a mark image indicating a mark which is a position for displaying the object is inputted, an imaging part which generates a plurality of input images by imaging at a plurality of time points, a filter part which performs filter processing to each of the input images, to generate a first image, an extraction part which extracts the feature point of the mark in the first image, a tracking part which tracks the feature point according to the change of the time point, a cut out part which cuts out a range including the mark from the first image, to generate a second image, a conversion matrix generation part which generates a conversion matrix on the basis of the second image and the mark image, and a display part which synthesizes the object with the input image on the basis of the conversion matrix and displays the resultant image.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing device, an image processing system and a program for AR.

A method of synthesizing and displaying an object prepared in advance with an captured image or the like by AR (Augmented Reality) technology is known.

For example, there is known a method in which an information processing device appropriately superimposes and displays a virtual object on an image showing a real space by AR. In this method, the information processing apparatus first divides the input image into semantic regions. Next, the information processing apparatus performs an optical flow on the input image of the immediately preceding frame and the input image of the current frame. In this way, by using the optical flow, the information processing device predicts the next movement of the real object. Then, the information processing device creates a position map centered on the display area of the virtual object. Next, the information processing device calculates the total score in the display area of the virtual object based on the importance map or the like. In this way, the information processing device determines the position where the virtual object is displayed. By performing the above processing, the information processing apparatus displays the virtual object more appropriately in consideration of the time change of the real object. Such a method and the like are known (see, for example, Patent Document 1 and the like).

Japanese Unexamined Patent Publication No. 2016-95579

However, it may be difficult to display the object with high accuracy by the method by the prior art. Specifically, in AR, the image processing device sets an object for a character, a figure, a symbol, a three-dimensional shape, a color, or a combination thereof (hereinafter, simply referred to as a “mark”) that serves as a mark for displaying the object in advance. Is combined and displayed. That is, the image processing device recognizes the mark by inputting an image indicating the mark in advance, and synthesizes and displays the object based on the position where the mark is recognized. On the other hand, in the conventional method, the object may not be displayed accurately with respect to the mark.

In view of the above problems, an object of the present invention is to provide an image processing device, an image processing system, a program, and the like for displaying an object with high accuracy in AR.

In view of the above problems, the image processing apparatus or the like according to the embodiment of the present invention is
A mark image input unit for inputting a mark image indicating a mark that is a position to display an object,
An imaging unit that captures images at multiple time points and generates multiple input images,
A filter unit that performs filtering processing on each of the input images to generate a first image,
An extraction unit that extracts the feature points of the mark in the first image,
A tracking unit that tracks the feature points with respect to the change at the time point,
A cutout portion that cuts out a range including the mark from the first image to generate a second image, and
A transformation matrix generator that generates a transformation matrix based on the second image and the marker image,
A display unit that synthesizes and displays the object on the input image based on the transformation matrix is included.

In AR, it is possible to provide an image processing device, an image processing system, a program, and the like for displaying an object with high accuracy.

It is a schematic diagram which shows the use example of an image processing apparatus. It is a block diagram which shows the hardware configuration example of an image processing apparatus. It is a flowchart which shows an example of the image processing method performed by an image processing apparatus. It is a figure which shows the processing result example of the image processing method. It is a figure which shows the example of a filter. It is a figure which shows the 1st modification of the filter processing. It is a figure which shows the 2nd modification of the filter processing. It is a figure which shows the extraction example of a feature point. It is a figure which shows the tracking example of a feature point. It is a figure which shows the cut-out example. It is a figure which shows the processing example which generates the transformation matrix. It is a figure which shows the example of comparison of the mark image and the conversion image, and the example of the comparison result. It is a figure which shows the generation example of an output image. It is a functional block diagram which shows the functional configuration example of an image processing apparatus.

Hereinafter, a specific example according to an embodiment of the present invention will be described with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicate description will be omitted.

<Overall configuration example>
FIG. 1 is a schematic view showing a usage example of an image processing device. Hereinafter, as illustrated, an example will be described in which the image processing device is a smartphone 10. The hardware configuration of the smartphone 10 and the like will be described later.

As in this example, the user UR uses the smartphone 10 to display the output image IMGV, which is an AR display, on the screen or the like. On the other hand, the user UR uses the imaging device of the smartphone 10 to image the subject 11 and the like. The imaging may be in the form of a still image by the operation of releasing the shutter, or in the form of a moving image for continuous imaging.

Hereinafter, a case where the subject 11 having a mark is imaged by the smartphone 10 as shown in the figure will be described as an example.

<Hardware configuration example of image processing device>
FIG. 2 is a block diagram showing a hardware configuration example of the image processing device. For example, as shown in the figure, the image processing device is hardware including a CPU (Central Processing Unit, hereinafter referred to as “CPU 10HW1”), a storage device 10HW2, an interface 10HW3, an input / output device 10HW4, a communication device 10HW5, an image pickup device 10HW6, and the like. It is a configuration.

The CPU 10HW1 is an example of an arithmetic unit and a control unit. That is, the CPU 10HW1 performs arithmetic operations, hardware resource control, and the like in order to execute processing based on a program or the like.

The storage device 10HW2 is an example of a main storage device, an auxiliary storage device, and the like.

The interface 10HW3 is connected to an external device, a recording medium, or the like by wire or wirelessly.

The input / output device 10HW4 is, for example, a touch panel or the like. That is, the input / output device 10HW4 is an input device that accepts operations from the user UR. The input / output device 10HW4 is an output device that outputs an image, a processing result, or the like.

The communication device 10HW5 is a device that transmits / receives data to / from an external device via a network by wire or wirelessly. For example, the communication device 10HW5 is an antenna, an electronic circuit, or the like.

The image pickup apparatus 10HW6 is, for example, an optical sensor, an optical component, or the like. Therefore, the image pickup device 10HW6 performs image pickup based on the operation of the user UR or the like, and generates an image.

The hardware configuration is not limited to the hardware configuration shown in the figure. For example, the image processing device may be configured to further include other hardware resources such as an arithmetic unit, a control device, a storage device, an input device, and an output device. On the other hand, the image pickup device, the input / output device, and the like may be an external device or the like.

Further, the image processing device may be another type of information processing device. For example, the image processing device may be a PC (Personal Computer), a tablet, or an information processing device such as a combination of a PC and a camera.

<Example of image processing method>
FIG. 3 is a flowchart showing an example of an image processing method performed by the image processing apparatus.

<Object image input example> (step S1)
In this step, the image processing apparatus inputs an object image. Hereinafter, the image showing the object is referred to as "object image IMGO".

The object image IMGO is an image showing an object to be combined and displayed at the time of output. The object is, for example, a 3D image, a character, a figure, a symbol, a three-dimensional shape, a color, or a combination thereof. That is, the object may be an object or the like that can be input as an image to the image processing device. Further, since the object is an object or the like to be displayed virtually, it does not have to exist, and may be a virtual object or the like generated by drawing software such as CAD (Computer Aided Design) or a graphic tool.

Further, the object image IMGO is input in a format such as an image file. For example, the object image IMGO is generated by imaging an object, generating an object with drawing software, or the like.

<Example of inputting a mark image> (step S2)
In this step, the image processing device inputs a marker image. Hereinafter, the image showing the mark is referred to as "mark image IMGT".

The landmark image IMGT is an image that serves as an index for determining the position, posture, range, and the like on which the objects are displayed when synthesizing the objects. The mark is, for example, a 3D image, a character, a figure, a symbol, a three-dimensional shape, a color, or a combination thereof. That is, the mark may be an object or the like that can be input as an image to the image processing device. Further, the mark does not need to exist as long as the position or the like can be grasped in the image, and may be a virtual object or the like generated by drawing software such as CAD or a graphic tool.

That is, the mark is a pattern or the like called a so-called marker.

Further, the mark image IMGT is input in a format such as an image file. For example, the mark image IMGT is generated by imaging an object on which the mark is drawn, or generating the mark by drawing software.

<Imaging example> (Step S3)
In this step, the image processing apparatus generates an image by imaging. Hereinafter, the image generated by imaging is referred to as "input image IMGI". Further, the input image IMGI is assumed to be a plurality of frames when the image processing device releases the shutter a plurality of times. Hereinafter, among the plurality of generated input image IMGIs, the latest frame may be referred to as the “current frame”. Then, a frame that is earlier than the current frame may be referred to as a "previous frame". In the following description, it is assumed that the frame rate is about 15 fps (frames per second) to about 60 fps. Then, in the following example, the case where the "previous frame" and the "current frame" are continuous frames will be described as an example.

As a measure against flicker, it is desirable that the imaging be performed at a frame rate deviated from the magnification of the frequency at which the illumination is turned on. Specifically, if the frame rate is 15 fps, 30 fps, 60 fps or the like, there is a possibility that the image is taken in synchronization with the frequency such as 60 Hz (hertz). Therefore, it is desirable that the frame rate is set to a value shifted by about 0.1 fps to 0.5 fps.

<Filter processing example> (step S4)
In this step, the image processing apparatus filters the input image. Details of the filtering process and examples of modifications will be described later.

Hereinafter, the image generated when the input image IMGI is filtered is referred to as "first image IMG1".

<Example of extracting feature points> (step S5)
In this step, the image processing apparatus performs a process of extracting feature points on the first image IMG1. The details of the process of extracting the feature points will be described later.

Hereinafter, the feature points extracted from the first image IMG1 by the process of extracting the feature points are referred to as "feature point PS". A plurality of feature point PSs may be extracted from the first image IMG1 of one frame.

Therefore, a generalized Hough transform or the like may be used to extract the feature points. That is, the feature points may be extracted not in units of points (that is, for one pixel) but in units of a collection of a plurality of feature points such as circles or lines. When extracted in such units, the image processing apparatus can extract feature points with higher accuracy.

<Example of tracking feature points> (step S6)
In this step, the image processor tracks the extracted feature points. The details of the process of tracking the feature points will be described later. Hereinafter, when the first image of the previous frame (hereinafter simply referred to as "previous frame image IMG11") and the first image of the current frame (hereinafter simply referred to as "current frame image IMG12") have the same feature point PS. Will be described as an example.

<Example of cutting out a range including a mark> (step S7)
In this step, the image processing apparatus cuts out a range including the mark to generate an image. Hereinafter, the image cut out and generated is referred to as "second image IMG2". The details of cutting out will be described later.

<Example of generating a transformation matrix based on the second image and the marker image> (step S8)
In this step, the image processing apparatus generates a transformation matrix based on the second image and the landmark image. A specific example of the transformation matrix and details of the process for generating the transformation matrix will be described later.

<Example of conversion of the second image based on the transformation matrix> (step S9)
In this step, the image processing apparatus transforms the second image based on the transformation matrix. Hereinafter, the image generated by the transformation based on the transformation matrix is referred to as "transformed image IMGC". The details of the conversion will be described later.

<Comparison example of mark image and converted image> (step S10)
In this step, the image processing apparatus compares the marker image with the converted image. The details of the process of comparing the mark image and the converted image will be described later.

<Example of determining whether or not there is a mark> (step S11)
In this step, the image processing apparatus determines whether or not there is a mark based on the comparison result of the mark image and the converted image.

Then, when it is determined that the mark is in the image (corresponds to YES in step S11), the image processing apparatus proceeds to step S12. On the other hand, if it is determined that the mark is not in the image (corresponds to NO in step S11), the image processing apparatus proceeds to step S3.

<Example of synthesizing and displaying an object at a mark of an input image> (step S12)
In this step, the image processing device synthesizes and displays the object at the mark portion of the input image. Hereinafter, the image showing the result of synthesizing the objects, that is, the output image is referred to as "output image IMGV". Specific examples of the output image IMGV and composition will be described in detail later.

<Example of determining whether or not to end> (step S13)
In this step, the image processing apparatus determines whether or not to terminate. That is, the image processing device determines whether or not to proceed to the next frame based on the setting, the switch, or the like.

Next, when it is determined to end (corresponding to YES in step S13), the image processing apparatus ends the illustrated image processing method. On the other hand, if it is determined that the process does not end (corresponds to NO in step S13), the image processing apparatus proceeds to step S3.

It should be noted that each step in the illustrated process does not have to be performed continuously. For example, the input of the object image, the input of the mark image, and the like are performed once, and when the image is saved, it may be reused thereafter. That is, steps S1 and S2 and the like may be omitted.

Further, the image processing method is not limited to the order of the illustrated processing. For example, steps S1 and S2 may be in parallel or in an order opposite to the order shown in the figure.

<Example of processing result of image processing method>
FIG. 4 is a diagram showing an example of processing results of the image processing method. When the image processing method shown in FIG. 3 is performed, the processing result is as shown in the figure, for example.

As shown in the figure, the output image IMGV is generated by synthesizing an object at a place where there is a mark in the input image IMGI.

First, let us take an example in which three subjects, a first subject TG1, a second subject TG2, and a third subject TG32, are captured in the input image IMGI.

Further, in this example, it is assumed that the mark image IMGT and the object image IMGO as shown in the drawings are input to the image processing apparatus in advance in steps S1 and S2. That is, in this example, the object TG41 shown by the object image IMGO is combined and displayed at the place where the mark TG31 indicated by the mark image IMGT is located.

In the input image IMGI, as shown in the figure, the object TG41 does not exist as a subject. Therefore, after recognizing the mark TG31, the image processing apparatus synthesizes the object TG41 by AR. In this way, the image processing device can display the object TG41 on the output image IMGV.

Specifically, in the case of the input image IMGI and the mark image IMGT as shown in the figure, when the image processing apparatus performs steps S4 to S10 and the like, the input image IMGI includes the mark. It can be recognized (corresponds to YES in step S11). Therefore, the image processing apparatus can synthesize the object TG 41 in accordance with the third subject TG 32 indicating the recognized mark TG 31 (step S12).

As shown in the figure, since the image processing device can accurately recognize the mark TG31, even if the input image IMGI has an object other than the mark such as the first subject TG1 and the second subject TG2, the third subject The object TG41 can be synthesized at a certain place of the TG32.

Further, the illustrated example is an example in which the posture and size of the mark TG31 and the third subject TG32 are different. Specifically, the third subject TG 32 appears larger in the input image IMGI than the mark TG 31 indicated by the mark image IMGT. Further, the third subject TG 32 is photographed in a posture having a so-called Pitch angle (in the figure, a rotation direction about the horizontal direction) with respect to the mark TG 31 indicated by the mark image IMGT.

As described above, the image processing apparatus can accurately recognize the mark TG 31 even when there is a so-called perspective on the mark image IMGT.

Further, since the posture and size of the third subject TG32 can be recognized, as shown in the figure, the image processing device converts the object TG41 according to the posture and size of the third subject TG32, and converts the object TG42. Can be displayed.

<Details of each procedure>
Each procedure in the image processing method is, for example, the following processing.

<Examples of filtering and filters used in filtering>
FIG. 5 is a diagram showing an example of a filter. In step S4 in the above image processing method, for example, a filter as shown in the figure is used.

The example shown is an example in which the filter is 3 × 3. For each cell of the filter, for example, a multiplier for multiplying each pixel value is set with a value as shown in the figure. Specifically, the image processing device has a pixel value of a central pixel (in the figure, a pixel at a position where a multiplier of "41" is set) with respect to each pixel value of the input image. Is multiplied by a multiplier of "41". Similarly, the image processing apparatus sets the pixel value of the pixels adjacent to the top, bottom, left, and right of the center pixel (in the figure, the pixels at the position where the multiplier of "-10" is set) to "-10". Calculate by multiplying by a multiplier. Then, the image processing device sums the calculation results calculated in this way. When such processing is performed on all the pixels in the image, that is, as in the so-called raster scan, the image processing apparatus can generate a first image having the effect of so-called sharpness.

The filter is not limited to the type shown in the figure. For example, a value other than the one shown in the figure may be used as the multiplier. That is, the intensity of sharpness may be adjusted by changing the value of the multiplier from the value shown in the figure. Further, the filtering process may be performed a plurality of times. That is, when emphasizing more, the filter processing may be performed twice or more by using a filter for sharpness or the like.

Further, the filter is not limited to the size of 3 × 3, and may be a filter having a large size of 5 × 5 or more.

<First modification of filtering>
FIG. 6 is a diagram showing a first modification of the filtering process. Step S4 in the above image processing method may be, for example, the process shown in the figure. As shown in the figure, in the first modification, the filter processing in step S4 is divided into two stages, the first filter processing and the second filter processing.

<First filter processing> (step S41)
In this step, the image processing apparatus performs filtering processing to reduce noise. Specifically, the image processing apparatus performs the first filter processing using, for example, a median filter or the like. The filter may be a filter that reduces noise. Therefore, the filter may be a low-pass filter or the like.

The input image IMGI generated by imaging often contains a component that causes noise. In processing such as pattern matching, if there are pixels that cause such noise, a phenomenon such as matching of noise pixels is likely to occur. Therefore, it causes deterioration of matching accuracy and the like. Therefore, if filter processing using a filter that reduces noise is performed, recognition accuracy such as pattern matching can be improved in the processing after the filter processing.

Hereinafter, the image generated by performing the first filter processing on the input image IMGI is referred to as "third image IMG3".

<Second filter processing> (step S42)
In this process, the image processing device performs a filter process for emphasizing the G component in the RGB color space. First, when the third image IMG3 is generated by the first filter processing, the third image IMG3 has an R component (becomes a red component) and a G component (becomes a green component) in the RGB color space. , And the B component (which becomes the blue component) can be divided into three planes.

Hereinafter, the plane showing the R component of the third image IMG3 is referred to as "R component IMG3R". Similarly, the plane showing the G component of the third image IMG3 is referred to as "G component IMG3G". Further, the plane showing the B component of the third image IMG3 is referred to as "B component IMG3B". The R component IMG3R, the G component IMG3G, and the B component IMG3B have, for example, the same data format as the image. Therefore, the R component IMG3R, the G component IMG3G, and the B component IMG3B can be filtered by the same method as other images.

It is desirable that the second filter treatment is performed on the G component IMG3G among the R component IMG3R, the G component IMG3G and the B component IMG3B.

In the first image IMG1 generated by the filter processing (step S4), after the filter processing, processing such as image comparison, processing for recognizing the subject appearing in the image, extraction and tracking of feature points is performed. In such processing, an edge component, a luminance component, or the like is often used. The luminance component (hereinafter referred to as “Y component”) is calculated from the RGB component as, for example, the following equation (1) or the following equation (2).

Y = 0.299 × R + 0.587 × G + 0.114 × B (1)

Y = 0.2126 × R + 0.7152 × G + 0.0722 × B (2)

As shown in the above equations (1) and (2), the Y component (the left-hand side "Y" in the above equations (1) and (2)) is a G component (above) in terms of the RGB component. In many cases, the ratio of the right-hand side “G” in the equation (1) and the above equation (2) is about 50% to 80%. That is, the G component is a component having a large influence on the Y component.

Therefore, in the second filter processing, the image processing apparatus performs a filter processing that emphasizes the G component IMG3G so that the Y component stands out in the first image IMG1. For example, the image processing apparatus performs processing using a sharpness filter or the like on the G component IMG3G. In this way, the image processing apparatus can generate the first image IMG1 in which the G component is emphasized.

In this way, with respect to the image generated based on the first image IMG1 in which the G component is emphasized or the first image IMG1 in which the G component is emphasized, image comparison, processing for recognizing the subject appearing in the image, and feature points When processing such as extraction and tracking is performed, the edge or feature point is clear, so that the accuracy of the processing can be improved.

<Second modification of filtering>
FIG. 7 is a diagram showing a second modification of the filtering process. Step S4 in the above image processing method may be, for example, the process shown in the figure. Compared with the first modification, the second modification is different in that the second filter processing is performed in the YUV color space. Hereinafter, the same configurations as those of the first modification will be described with the same reference numerals, omitting duplicate description, and focusing on different points.

As shown in the figure, in the second modification, the third image IMG3 has a Y component (which is a luminance component) and a U component (the R component minus the Y component, that is, a Cr component) in the YUV color space. It can be divided into three planes: a "Cr component") and a V component (a so-called Cb component obtained by removing the Y component from the B component. Hereinafter referred to as a "Cb component").

Therefore, when the input image IMGI or the like is in the data format of the RGB color space, the image processing apparatus performs a process of converting the RGB data into the YUV data. For example, the conversion formulas are PAL (phase alternation line), SECAM, ITU-R BT. 601. ITU-R BT. 709 (1250/50/2: 1) or ITU-R BT. A conversion formula or the like defined by 709 (1125/60/2: 1) or the like is used.

Hereinafter, the plane showing the Y component of the third image IMG3 is referred to as "Y component IMG3Y". Similarly, the plane showing the Cr component of the third image IMG3 is referred to as "Cr component IMG3CR". Further, the plane showing the Cb component of the third image IMG3 is referred to as "Cb component IMG3CB". The Y component IMG3Y, Cr component IMG3CR, and Cb component IMG3CB have, for example, the same data format as the image. Therefore, the Y component IMG3Y, the Cr component IMG3CR, and the Cb component IMG3CB can be filtered in the same manner as other images.

The second filter treatment is preferably performed on the Y component IMG3Y among the Y component IMG3Y, the Cr component IMG3CR, and the Cb component IMG3CB.

Similar to the first modification, even if the first image IMG1 is generated by the filtering process (step S4) of the second modification, in the subsequent stage after the filtering, the image is compared with the first image IMG1. Processing such as recognition of the subject in the image, extraction of feature points, and tracking is performed. Therefore, it is desirable that the Y component is emphasized in the first image IMG1.

Therefore, when the filter processing as shown in the figure is performed, the image processing apparatus can generate the first image IMG1 in which the Y component is emphasized.

In this way, with respect to the image generated based on the first image IMG1 with the Y component emphasized or the first image IMG1 with the Y component emphasized, image comparison, processing for recognizing the subject appearing in the image, and feature points When processing such as extraction and tracking is performed, the edge or feature point is clear, so that the accuracy of the processing can be improved.

Further, if the data format is such that the U component, the V component, or both components are smaller than the data of the Y component, such as YUV422 or YUV411, the data capacity can be reduced without significantly reducing the image quality.

The U component and the V component often have low spatial resolution, that is, even if the data is thinned out from the Y component, the image quality to the human eye is not significantly affected. Therefore, the image processing apparatus may further perform a process using a low-pass filter or the like on the U component, the V component, or both components to reduce noise.

The YUV color space may be a YCbCr color space, a YPbPr color space, a Lab color space, or the like. That is, the component of YUV may be a component of YCbCr, YPbPr or Lab.

<Example of extracting feature points>
FIG. 8 is a diagram showing an example of extracting feature points. Step S5 in the above image processing method is, for example, a process as shown in the figure.

Extraction of feature points is realized by, for example, KPM (Key Point Matching) or the like. That is, if there is an edge or pixel value or the like that is the same as the image input in advance (hereinafter referred to as "reference image"), the image processing apparatus extracts it as a feature point. Then, as shown in the figure, when the first image IMG1 is subjected to processing such as KPM, the feature point PS is extracted. Further, when the feature points are extracted, the image processing apparatus can recognize each of the extracted feature point PS in the subsequent processing. Therefore, when the feature point PS is extracted, the image processing apparatus can recognize the subject reflected in the first image IMG1 or the like such as a mark based on the position and configuration of the feature point PS. Then, based on the extraction result of the feature point PS, the presence / absence of a mark or the position of the mark in the image can be recognized in the subsequent processing.

It is desirable that the feature points are extracted by an algorithm of AKAZE (Accelerated KAZE Features) or ORB (Oriented FAST and Rotated BRIEF).

According to the AKAZE algorithm, the image processing apparatus can extract in the low frequency region, the high frequency region, and the like with higher accuracy than the ORB and the like.

In the ORB algorithm, the image processing apparatus has an image to be processed for extracting feature points, that is, the first image IMG1 has movement, rotation, scaling, or a combination thereof with respect to the reference image. However, it can be extracted with high accuracy. That is, by using the ORB algorithm, the image processing apparatus can ensure high robustness against movement, rotation, scaling, and the like. Further, with the ORB algorithm, the image processing apparatus can process at high speed.

The feature point PS may be extracted by a method other than the above. For example, the feature point PS may be extracted by a filter or the like that extracts edges.

<Example of tracking feature points>
FIG. 9 is a diagram showing an example of tracking feature points. Step S6 in the above image processing method is, for example, a process as shown in the figure.

Tracking of feature point PS is realized by, for example, optical flow. Hereinafter, as shown in the figure, it is assumed that the first image is captured in the order of the previous frame image IMG11, the current frame image IMG12, and the next frame image of the current frame (hereinafter referred to as "next frame image IMG13").

Using optical flow, as shown, it is possible to track how the feature point PS moves between the previous frame image IMG11, the current frame image IMG12, and the next frame image IMG13. Therefore, when the feature point PS of the mark is tracked, the image processing device can track the place that becomes the mark, so that the object can be accurately synthesized at the position of the mark.

For example, the optical flow is realized by the Lucas-Kanade method or the like.

It should be noted that it may be determined whether or not to output the object in the output image based on the processing result of the optical flow. Specifically, based on the processing result of the optical flow, there is a case where the subject having the feature point PS of the mark moves out of the range indicated by the image. This is the case when the so-called subject is out of frame. In such a case, the image processing apparatus may proceed to the next frame (step S3) by omitting the processing after step S7 such as outputting an object based on the processing result of the optical flow.

<Cutout example>
FIG. 10 is a diagram showing a cutout example. Step S7 in the above image processing method is, for example, a process as shown in the figure. Hereinafter, a case where the same subject as in FIG. 4 is captured and the mark image is also the same image will be described as an example. In the illustrated example, the result of the filter processing is not reflected in the drawing in the first image IMG1, and an image similar to the input image IMG shown in FIG. 4 will be described as the first image IMG1.

In the cropping process, the image processing apparatus sets a range including the third subject TG32, that is, the feature point of the mark among the subjects of the first subject TG1, the second subject TG2, and the third subject TG32 reflected in the first image IMG1. It is cut out to generate a second image IMG2.

That is, the second image IMG2 is generated, for example, by cutting out an image including all the feature points of the mark (a margin may be set) or a size including all the pixels of the subject recognized as the mark. ..

As shown in the figure, the size of the second image IMG2 is smaller than that of the first image IMG1 and the like. That is, the second image IMG2 is an image having fewer pixels than the first image IMG1 and the like. Further, the second image IMG2 is an image in which the first subject TG1 or the like is not captured as compared with the first image IMG1 or the like. Therefore, when the second image IMG2 is used in the subsequent processing, the image processing apparatus has a smaller number of pixels and the like as compared with the case where the first image IMG1 and the like are used, so that the calculation cost can be reduced or the calculation cost can be reduced or at high speed. Matching processing can be performed.

<Process to generate transformation matrix and conversion example based on transformation matrix>
FIG. 11 is a diagram showing a processing example for generating a transformation matrix. Steps S8 and S9 in the above image processing method are, for example, processes as shown in the figure.

For example, the transformation matrix MX1 is a matrix of 3 rows and 3 columns as shown in the figure. Then, as shown in the figure, the transformation matrix MX1 is a matrix that projects and transforms the target image.

When the transformation using the transformation matrix MX1 is performed, the second image IMG2 becomes the transformed image IMGC as shown in the figure. As shown in the figure, the converted image IMGC is an image converted so that the subject (that is, the mark) indicated by the mark image IMGT and the subject have substantially the same size and posture.

Hereinafter, it is assumed that three feature points of the first feature point PS1, the second feature point PS2, and the third feature point PS3 are extracted from the second image IMG2. It is assumed that the marker image IMGT also has three feature points, that is, the first feature point PS1, the second feature point PS2, and the third feature point PS3.

Therefore, which pixel on the second image IMG2 the first feature point PS1, the second feature point PS2, and the third feature point PS3 in the landmark image IMGT are located is determined in the second image IMG2 as the first feature point. It can be specified by searching for PS1, the second feature point PS2, and the third feature point PS3, respectively. Therefore, when the image processing apparatus converts the first feature point PS1, the second feature point PS2, and the third feature point PS3 on the second image IMG2, the first feature point PS1 and the second feature point PS1 in the landmark image IMGT are converted. It is possible to calculate whether the positions of the points PS2 and the third feature point PS3 can be converted to almost the same positions. Based on such a calculation result, the image processing apparatus can generate the transformation matrix MX1. That is, the image processing apparatus generates the transformation matrix MX1 by so-called homography estimation or the like.

Next, based on the generated transformation matrix MX1, the image processing apparatus converts the second image IMG2 to generate the converted image IMGC.

In this way, the image processing apparatus projects and transforms the second image IMG2 by the transformation matrix MX1. When such conversion is performed, as shown in the figure, the subject reflected in the second image IMG2 can be stretched to obtain a subject in a state similar to that of the landmark image IMGT. In this way, in the process of comparison and the like, the mark reflected on the second image IMG2 is in a state of having the same position, size, posture and the like as the mark image IMGT. Therefore, there is not much scaling, rotation, parsing, etc. with respect to the mark indicated by the mark image IMGT, and matching accuracy using an algorithm such as ORB or AKAZE can be improved.

<Comparison example of mark image and converted image>
FIG. 12 is a diagram showing a comparison example of the mark image and the converted image and an example of the comparison result. Step S10 in the above image processing method is, for example, a process as shown in the figure.

As shown, the image processing apparatus compares the landmark image IMGT with the converted image IMGC. When compared in this way, the image processing apparatus can determine whether or not the converted image IMGC is a marker image.

Hereinafter, as shown in the figure, it is assumed that the converted image IMGC has four feature points, such as a first point SP1, a second point SP2, a third point SP3, and a fourth point SP4. Then, it is assumed that the coordinates of the first point SP1, the second point SP2, the third point SP3, and the fourth point SP4 are specified in the feature point image IMGS for processing such as comparison.

<Example of combining an object with an input image to generate an output image>
FIG. 13 is a diagram showing an example of generating an output image. Hereinafter, for the sake of explanation, a case where a feature point image IMGS in which the feature points and the like are illustrated will be used will be described as an example. However, unlike the feature point image IMGS, the image used for compositing does not need to show the first point SP1, the second point SP2, the third point SP3, and the fourth point SP4, and the second image IMG2 or the like is also used. Good.

First, the image processing apparatus can calculate the inverse transformation matrix MX2, which is an example of the inverse matrix, by calculating the inverse matrix of the transformation matrix.

Next, the image processing apparatus projects and transforms the feature point image IMGS using the inverse transformation matrix MX2. In this way, as shown in the figure, the conversion feature point image IMGS2 can be generated.

Subsequently, the image processing apparatus projects and transforms the object image IMGO using the inverse transformation matrix MX2. In this way, the converted object image IMGO2 can be generated as shown in the figure.

As described above, when both transformations are performed using the inverse transformation matrix MX2 which is the same matrix, the transformation feature point image IMGS2 and the transformation object image IMGO2 are generated by the same transformation. Therefore, in the conversion feature point image IMGS2 and the conversion object image IMGO2, the marks and objects tend to have the same size and orientation. Therefore, the image processor can synthesize the objects so that they have a more natural shape with respect to the landmarks.

When combined as described above, the image processing apparatus can display the output image IMGV in which the object is accurately combined at the place where the mark is located, as shown in FIG. 4, for example.

<Function configuration example>
FIG. 14 is a functional block diagram showing a functional configuration example of the image processing device. For example, as shown in the figure, the image processing apparatus includes a mark image input unit 10F1, an imaging unit 10F2, a filter unit 10F3, an extraction unit 10F4, a tracking unit 10F5, a cutting unit 10F6, and a transformation matrix generation unit 10F7. It is a functional configuration including the display unit 10F8.

Further, as shown in the drawing, it is desirable that the filter unit 10F3 has a functional configuration including the first filter unit 10F31, the second filter unit 10F32, and the like.

Further, as shown in the figure, it is desirable that the image processing apparatus has a functional configuration further including a re-extraction unit 10F9, a calculation unit 10F10, and the like. Hereinafter, a functional configuration as shown will be described as an example.

The mark image input unit 10F1 performs a mark image input procedure for inputting a mark image indicating a mark that serves as a position for displaying an object. For example, the mark image input unit 10F1 is realized by the interface 10HW3 or the like.

The imaging unit 10F2 performs an imaging procedure of imaging at a plurality of time points to generate a plurality of input images. For example, the image pickup unit 10F2 is realized by an image pickup device 10HW6 or the like.

The filter unit 10F3 performs a filter procedure for generating a first image by performing a filter process on each input image. For example, the filter unit 10F3 is realized by the CPU 10HW1 or the like.

The first filter unit 10F31 performs the first filter procedure using a median filter or the like on the input image.

The second filter unit 10F32 performs a second filter processing procedure using a sharpness filter that emphasizes the G component in the RGB color space for the third image generated by the first filter unit 10F31 performing the first filter processing procedure. Do. Alternatively, the second filter unit 10F32 uses a sharpness filter that emphasizes the Y component in the YUV color space for the third image generated by the first filter unit 10F31 performing the first filter processing procedure. Perform the procedure.

The extraction unit 10F4 performs an extraction procedure for extracting feature points of marks in the first image. For example, the extraction unit 10F4 is realized by the CPU 10HW1 or the like.

The tracking unit 10F5 performs a tracking procedure for tracking the feature points extracted by the extraction unit 10F4 with respect to the change at the time point. For example, the tracking unit 10F5 is realized by the CPU 10HW1 or the like.

The cutout portion 10F6 performs a cutting procedure of cutting out a range including a mark from the first image and generating a second image. For example, the cutout portion 10F6 is realized by the CPU 10HW1 or the like.

The transformation matrix generation unit 10F7 performs a transformation matrix generation procedure for generating a transformation matrix based on the second image and the mark image. For example, the transformation matrix generation unit 10F7 is realized by the CPU 10HW1 or the like.

The display unit 10F8 performs a display procedure of synthesizing and displaying an object with an input image based on a transformation matrix. For example, the display unit 10F8 is realized by an input / output device 10HW4 or the like.

The re-extraction unit 10F9 performs a re-extraction procedure for extracting feature points in the second image. For example, the re-extraction unit 10F9 is realized by the CPU 10HW1 or the like.

The calculation unit 10F10 performs a calculation procedure for calculating the inclination of the imaging unit 10F2 and the distance to the mark based on the feature points extracted by the re-extraction unit 10F9. For example, the calculation unit 10F10 is realized by the CPU 10HW1 or the like.

<Summary>
First, the image processing device inputs an object image indicating an object to be displayed by AR. Further, the image processing device inputs a mark image indicating a mark such as a position for displaying the object. In this way, when the object image and the mark image are input in advance, the image processing device can recognize the mark and display the object.

Next, the image processing device generates an input image to be combined with the object by imaging.

For example, the filter unit 10F3 performs filter processing by the filter or the like shown in FIG. The filter unit 10F3 may be filtered by the first filter unit 10F31, the second filter unit 10F32, or the like to emphasize the G component or the Y component, for example, as shown in FIG. 6 or FIG.

Then, when the first image is generated by the filter processing by the filter unit 10F3, the image processing apparatus extracts the feature points of the marks as shown in FIG. 8, for example. Subsequently, the image processing apparatus tracks the extracted feature points between frames, for example, as shown in FIG.

Next, the image processing apparatus cuts out a range including a mark in the first image and generates a second image, for example, as shown in FIG. In this way, a small size image can be generated. Therefore, in the subsequent processing, the number of pixels to be calculated can be reduced, and the calculation cost can be reduced.

Further, the image processing apparatus generates a transformation matrix by using the same feature points and the like in the mark image and the second image, for example, as shown in FIG. When the transformation matrix is generated in this way, the image processing apparatus performs the transformation by the transformation matrix, and as shown in FIG. 12, the subject reflected in the transformed image and the marker indicated by the marker image have the same size and orientation. And so on. Therefore, in the input image, even if the image pickup device is tilted or the subject having a mark is small and appears in the input image due to a distance from the subject, the image processing device accurately corrects the mark. I can recognize it well.

Further, when the inverse matrix is generated based on the transformation matrix, the image processing apparatus can synthesize an object with the input image as shown in FIG. 13, and can display a natural AR.

In addition, the process may be performed separately from the synthesis shown in FIG. 13 and the like by re-extraction and calculation. For example, in the re-extraction, the feature points are extracted by the same method as the extraction unit 10F4. However, the re-extraction targets an image different from the extraction unit 10F4. Then, the calculation unit 10F10 estimates the inclination and the distance of the imaging unit 10F2 by, for example, PnP. That is, the calculation unit 10F10 calculates "R" and "t" in the so-called camera parameters.

That is, "R", that is, "Roll angle", "Pitch angle" (sometimes referred to as "Tilt angle") and "Yaw angle" (sometimes referred to as "Pan angle") with respect to the subject possessed by the subject. ) Etc., if there is a result of calculating how much the angle is, it is possible to estimate what kind of inclination the image processing device will take when the mark is the input image.

Similarly, if there is a result of calculating "t", that is, how far the subject is from the subject, what size does the image processing device appear in the input image as the mark? Can be estimated.

In this way, the image processing device calculates the relative relationship between the inclination and the distance between the target subject and the image pickup device by PnP or the like. If the slope and distance can be calculated, the calculation method is not limited to PnP.

For example, in this way, it is desirable to calculate the inclination of the image pickup unit 10F2 with respect to the mark in the input image and the distance of the image pickup unit 10F2 from the mark. With such a calculation result, the image processing apparatus can synthesize objects with higher accuracy.

<Other Embodiments>
It should be noted that each device of the image processing device, the image processing system, and the like or an external device to be connected does not have to be realized by one device. That is, each device may be composed of a plurality of devices. For example, an image processing system may have a plurality of information processing devices or imaging devices, and each processing may be performed in a distributed, parallel, or redundant manner.

It should be noted that all or part of each process according to the present invention is described in a low-level language such as an assembler or a high-level language such as an object-oriented language, and may be realized by a program for causing a computer to execute an image processing method. Good. That is, the program is a computer program for causing a computer such as an image processing device and an image processing system to execute each process.

Therefore, when the image processing method is executed based on the program, the arithmetic unit and the control device of the computer perform the arithmetic and control based on the program in order to execute each process. In addition, the storage device of the computer stores the data used for the processing based on the program in order to execute each processing.

In addition, the program can be recorded and distributed on a computer-readable recording medium. The recording medium is a medium such as a magnetic tape, a flash memory, an optical disk, a magneto-optical disk, or a magnetic disk. In addition, the program can be distributed over telecommunication lines.

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above. Therefore, the embodiments can be modified or modified in various ways within the scope of the gist of the invention described in the claims.

10 Smartphone 10F1 Mark image input unit 10F10 Calculation unit 10F2 Imaging unit 10F3 Filter unit 10F31 First filter unit 10F32 Second filter unit 10F4 Extraction unit 10F5 Tracking unit 10F6 Cutout unit 10F7 Conversion matrix generation unit 10F8 Display unit 10F9 Re-extraction unit 10HW1 CPU
10HW2 Storage device 10HW3 Interface 10HW4 Input / output device 10HW5 Communication device 10HW6 Imaging device 11 Subject IMG1 First image IMG2 Second image IMG3 Third image IMG3Y Y component IMG3CB Cb component IMG3CR Cr component IMG3R R component IMG3G G component IMG3B Image IMG12 Current frame image IMG13 Next frame image IMGC Converted image IMGI Input image IMGO Object image IMGO2 Converted object image IMGS Feature point image IMGS2 Converted feature point image IMGT Marking image IMGV Output image MX1 Transformation matrix MX2 Inverse transformation matrix PS Feature point PS1 1st Feature point PS2 2nd feature point PS3 3rd feature point SP1 1st point SP2 2nd point SP3 3rd point SP4 4th point TG1 1st subject TG2 2nd subject TG31 Marker TG32 3rd subject TG41 Object TG42 Conversion object UR user

In view of the above problems, the image processing apparatus or the like according to the embodiment of the present invention is
A mark image input unit for inputting a mark image indicating a mark that is a position to display an object,
An imaging unit that captures images at multiple time points and generates multiple input images,
A filter unit that performs filtering processing on each of the input images to generate a first image,
An extraction unit that extracts the feature points of the mark in the first image,
A tracking unit that tracks the feature points with respect to the change at the time point,
A cutout portion that cuts out a range including the mark from the first image to generate a second image, and
A transformation matrix generator that generates a transformation matrix based on the second image and the marker image,
Based on the transformation matrix, seen including a display unit for displaying by combining the object to the input image,
Based on the transformation matrix, the second image is transformed to generate a transformed image.
When the mark image and the converted image are compared and it is determined that the converted image has the mark, the display unit converts both the object and the mark by the inverse matrix of the transformation matrix and synthesizes them .

Claims (9)

A mark image input unit for inputting a mark image indicating a mark that is a position to display an object,
An imaging unit that captures images at multiple time points and generates multiple input images,
A filter unit that performs filtering processing on each of the input images to generate a first image,
An extraction unit that extracts the feature points of the mark in the first image,
A tracking unit that tracks the feature points with respect to the change at the time point,
A cutout portion that cuts out a range including the mark from the first image to generate a second image, and
A transformation matrix generator that generates a transformation matrix based on the second image and the marker image,
An image processing device including a display unit that synthesizes and displays the object on the input image based on the transformation matrix. The filter unit
The first filter processing using a median filter on the input image and
The first image is generated by performing a second filter process using a sharpness filter that emphasizes the G component in the RGB color space on the third image generated by the first filter process.
The extraction unit
The image processing apparatus according to claim 1, wherein the G component is extracted based on the first image in which the G component is emphasized. The filter unit
The first filter processing using a median filter on the input image and
The first image is generated by performing a second filter process using a sharpness filter that emphasizes the Y component in the YUV color space on the third image generated by the first filter process.
The extraction unit
The image processing apparatus according to claim 1, wherein the Y component is emphasized and extracted based on the first image. The extraction unit
The image processing apparatus according to any one of claims 1 to 3, which is extracted by an algorithm of AKAZE or ORB. Based on the transformation matrix, the second image is transformed to generate a transformed image.
When the mark image and the converted image are compared and it is determined that the converted image has the mark, the display unit claims that both the object and the mark are converted by the inverse matrix of the conversion matrix and synthesized. Item 2. The image processing apparatus according to any one of Items 1 to 4. The transformation matrix is
The image processing apparatus according to claim 5, wherein the feature points in the second image are a matrix that is converted so as to be at the same position as the feature points in the mark image. A re-extraction unit that extracts feature points in the second image,
It further includes a calculation unit that calculates the inclination of the imaging unit and the distance to the mark based on the feature points extracted by the re-extraction unit.
The image processing apparatus according to any one of claims 1 to 6, wherein the display unit synthesizes the object based on the inclination and the distance. A mark image input unit for inputting a mark image indicating a mark that is a position to display an object,
An imaging unit that captures images at multiple time points and generates multiple input images,
A filter unit that performs filtering processing on each of the input images to generate a first image,
An extraction unit that extracts the feature points of the mark in the first image,
A tracking unit that tracks the feature points with respect to the change at the time point,
A cutout portion that cuts out a range including the mark from the first image to generate a second image, and
A transformation matrix generator that generates a transformation matrix based on the second image and the marker image,
An image processing system including a display unit that synthesizes and displays the object on the input image based on the transformation matrix. A program for causing an image processing device to execute an image processing method.
The mark image input procedure for inputting the mark image indicating the mark at which the image processing device displays the object, and
An imaging procedure in which an image processing device captures images at multiple time points and generates multiple input images.
A filter procedure in which the image processing device performs filtering processing on each of the input images to generate a first image, and
An extraction procedure in which the image processing device extracts the feature points of the mark in the first image, and
A tracking procedure in which the image processing device tracks the feature points with respect to the change at the time point.
An image processing device cuts out a range including the mark from the first image to generate a second image, and a cutting procedure.
A transformation matrix generation procedure in which the image processing apparatus generates a transformation matrix based on the second image and the landmark image, and
A program for causing an image processing apparatus to execute a display procedure of synthesizing and displaying the object on the input image based on the transformation matrix.