JP2019164620A - Determination device, determination method, and determination program - Google Patents

Abstract

To easily generate content through use of marker-less AR.SOLUTION: An extraction unit 151 extracts a feature point from a predetermined region within an image captured by an imaging unit 13. A calculation unit 152 calculates an angle between a feature surface which is a virtual surface generated on the basis of the feature point and a horizontal surface with respect to an imaging surface of the imaging unit 13. A determination unit 153 determines whether or not a predetermined condition concerning availability of the image in the predetermined region captured by the imaging unit 13 as a marker of marker-less AR is satisfied, on the basis of the feature point and the angle.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a determination device, a determination method, and a determination program.

  Conventionally, an AR (Augmented Reality) technique for recognizing a predetermined target from an image and displaying an object superimposed on the image is known. In addition, the AR technology methods include a marker AR and a markerless AR.

  The marker AR is a method that uses a specific recognition target created for the purpose of being recognized by a computer, such as a QR (Quick Response) code (registered trademark). On the other hand, the markerless AR is a method in which an arbitrary image set in advance is used as a recognition target without using a specific recognition target such as a QR code. Also, tools for creating content using markerless AR are known (see, for example, Non-Patent Documents 1 and 2).

ptc, “Vuforia”, [online], [Search March 6, 2018], Internet (https://developer.vuforia.com/) CYBERNET, “SmartAR”, [online], [Search March 6, 2018], Internet (http://www.cybernet.co.jp/ar-vr/products/smartar/)

  However, the conventional technology has a problem that it may be difficult to easily create content using the markerless AR. For example, when creating content using markerless AR, it is necessary to prepare an image suitable for a recognition target in order to improve recognition accuracy. Here, when a conventional tool is used, whether or not the prepared image is suitable for the recognition target is determined after the image is taken into the tool. For this reason, in order to prepare an image suitable for a recognition target before importing into a tool, a certain level of knowledge and technology is required. Especially for beginners, it is difficult to create content using markerless AR. It may become.

  The determination apparatus according to the embodiment includes an extraction unit, a calculation unit, and a determination unit. The extraction unit extracts a feature point from a predetermined region in the image captured by the imaging unit that captures an image. The calculation unit calculates an angle between a feature plane and a horizontal plane, which are virtual planes generated based on the feature points, and an imaging plane of the imaging unit. The determination unit determines whether or not a predetermined condition regarding availability as a markerless AR marker of an image of a predetermined region captured by the imaging unit is satisfied based on the feature point and the angle.

FIG. 1 is a diagram for explaining the outline of the determination apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of the configuration of the determination apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an example of a screen of the determination apparatus according to the first embodiment. FIG. 4 is a diagram illustrating an example of a screen of the determination apparatus according to the first embodiment. FIG. 5 is a diagram illustrating an example of a screen of the determination apparatus according to the first embodiment. FIG. 6 is a diagram illustrating an example of a screen of the determination apparatus according to the first embodiment. FIG. 7 is a diagram illustrating an example of a screen of the determination apparatus according to the first embodiment. FIG. 8 is a diagram illustrating an example of a screen of the determination apparatus according to the first embodiment. FIG. 9 is a flowchart showing a flow of processing of the determination apparatus according to the first embodiment.

  Hereinafter, embodiments of a determination device, a determination method, and a determination program according to the present application will be described in detail with reference to the drawings. In addition, this invention is not limited by embodiment described below. In the following description, an image to be recognized in the markerless AR may be referred to as a “marker”.

(First embodiment)
First, the outline | summary of the determination apparatus of this embodiment is demonstrated using FIG. FIG. 1 is a diagram for explaining the outline of the determination apparatus according to the first embodiment. As shown in FIG. 1, the determination apparatus 10 is possessed by a user 10a. For example, the determination device 10 is a tablet terminal, a smartphone, or the like.

  Further, the determination apparatus 10 can display an image captured using the imaging unit provided in the determination apparatus 10 on the display unit 12. Here, it is assumed that the determination apparatus 10 includes a camera as an imaging unit. In the example of FIG. 1, the determination apparatus 10 captures a scene around the facility 20. The imaging target of the determination device 10 is not limited to the facility 20 as long as it includes a markerless AR recognition target, and may be an artificial object such as a server device, a control panel, or a printer, or a natural object. May be.

  The determination apparatus 10 performs a determination process for determining whether or not a predetermined area in the captured image can be used as a marker, and presents the determination result to the user 10a. As shown in FIG. 1, the determination apparatus 10 receives an operation of the user 10a through each button. Moreover, the determination apparatus 10 superimposes and displays an object for presenting information to the user 10a on the captured image.

  First, when the start button 121b is pressed, the determination apparatus 10 starts processing. Here, the determination apparatus 10 receives designation of an area to be used as a marker from the user 10a. For example, when the user 10a designates two points on the captured image, the determination apparatus 10 sets a rectangular area having the two points as the opposite vertices as an area to be used as a marker, and displays a frame 122a indicating the area. When the reset button 121a is pressed, the designated area is reset.

  Next, when the feature point extraction button 121c is pressed, the determination apparatus 10 extracts feature points in the region. Note that the determination apparatus 10 may simultaneously extract feature points outside the region. For example, the determination apparatus 10 extracts an edge portion of the image as a feature point. In this case, the corners and line portions of the object, and portions where the difference in brightness and color clearly appear are more easily extracted as feature points. Further, it is desirable that a plurality of feature points exist on the same plane as an image for recognition. Here, when the sufficient feature points cannot be extracted from the area, the determination apparatus 10 displays a message that prompts the user to specify the area again.

  Moreover, the determination apparatus 10 displays the extracted feature points as points 122b. Further, when a part or all of the frame 122a protrudes from the imaging range of the camera of the determination apparatus 10, a message indicating that a part or all of the area is not within the imaging range is displayed.

  Furthermore, the determination apparatus 10 performs a process for determining whether the camera posture is correct. When the camera is horizontal and the imaging plane of the camera is parallel to the feature plane that is a virtual plane generated from the extracted feature points, the determination apparatus 10 determines that the posture of the camera is correct.

  The determination device 10 calculates an angle for determining whether or not the camera is horizontal and the feature surface and the imaging surface of the camera are parallel. Specifically, the determination apparatus 10 calculates an angle between the feature plane and the horizontal plane and the imaging plane of the camera. In addition, the determination apparatus 10 can calculate the angle between the horizontal plane of the determination apparatus 10 using a gyro sensor. Further, the fact that the feature plane and the imaging plane of the camera are parallel is the same as the fact that the feature plane and the optical axis of the camera are perpendicular.

  Then, when the calculated angle is within a predetermined range, the determination device 10 controls the capture button 121d to be operable. In addition, when the calculated angle is not within the predetermined range, the determination apparatus 10 controls the capture button 121d so that it cannot be operated, and corrects the posture of the camera so that the angle is within a range that can be regarded as horizontal or vertical. Display a prompt message.

  Moreover, the determination apparatus 10 displays the grid 122c on the feature plane. Moreover, the determination apparatus 10 displays the mark 122d on a virtual imaging surface. Moreover, the determination apparatus 10 displays the mark 122e on the feature surface. At this time, the determination apparatus 10 displays the mark 122d and the mark 122e so as to overlap when the virtual imaging surface and the feature surface are parallel to each other. Further, the distance between the mark 122d and the mark 122e decreases as the virtual imaging surface and the feature surface become parallel. The marks 122d and 122e are examples of the first mark and the second mark, respectively.

  When the capture button 121d is pressed, the determination device 10 acquires an image captured by the camera as a marker image. At this time, the determination apparatus 10 further acquires an image with an expanded imaging range as an editing image. The determination apparatus 10 may display a message instructing the user 10a to pull the camera in order to acquire an image with an expanded imaging range. The marker image and the editing image are examples of the first image and the second image, respectively.

  For example, in the example of FIG. 1, the frame 122a is displayed up in the marker image, and both the facility 20 and the object 30 are displayed in the editing image. The marker image is used to generate a markerless AR marker. On the other hand, the editing image is used as a mark when the AR object is arranged with respect to the marker.

(Configuration of the determination apparatus according to the first embodiment)
The configuration of the determination apparatus 10 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of the configuration of the determination apparatus according to the first embodiment. As illustrated in FIG. 2, the determination device 10 includes an input unit 11, a display unit 12, an imaging unit 13, a storage unit 14, and a control unit 15.

  The input unit 11 receives data input from the user. The input unit 11 is an input device such as a mouse or a keyboard, for example. The display unit 12 is a display device such as a display that displays a screen. The input unit 11 and the display unit 12 may be a touch panel display. The imaging unit 13 captures an image. For example, the imaging unit 13 is a camera.

  The storage unit 14 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disk, a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non Volatile Static Random Access Memory). The storage unit 14 stores an OS (Operating System) and a program executed by the determination apparatus 10. Furthermore, the memory | storage part 14 memorize | stores the various information used by execution of a program.

  The control unit 15 controls the determination device 10. The control unit 15 is a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. For example, the control unit 15 functions as an extraction unit 151, a calculation unit 152, a determination unit 153, a setting unit 154, a display control unit 155, and an acquisition unit 156.

  The extraction unit 151 extracts feature points from a predetermined region in the image captured by the imaging unit 13. The extraction unit 151 can extract the edge portion of the image as a feature point. In addition, the extraction unit 151 may extract the feature points of the area specified by the user, or may extract the feature points of the entire area of the image.

  The calculation unit 152 calculates an angle between a feature plane and a horizontal plane that are virtual planes generated based on the feature points and the imaging plane of the imaging unit 13. First, the calculation unit 152 identifies the imaging surface of the imaging unit 13 using Euclidean restoration. Furthermore, the calculation unit 152 specifies a feature plane including the region by back projection. Then, the calculation unit 152 calculates an angle between the identified feature plane and the imaging plane. In addition, the angle between the horizontal plane and the imaging plane can be calculated using the calculation unit 152 gyro sensor.

  The determination unit 153 determines, based on the feature point and the angle, whether or not a predetermined condition regarding the availability of the image of the predetermined area captured by the imaging unit 13 as the markerless AR marker is satisfied. The determination unit 153 determines that the predetermined condition is not satisfied when at least a part of the predetermined region set by the setting unit 154 is out of the imaging range of the imaging unit 13.

  The determination unit 153 determines that the predetermined condition is not satisfied when the number of feature points included in the region is equal to or smaller than the predetermined threshold value or when the angle is not within the predetermined range. For example, the determination unit 153 determines that the condition is not satisfied when the number of feature points included in the region is five or less. For example, the determination unit 153 determines that the condition is not satisfied when the angle between the feature surface and the imaging surface is not in the range of −2 ° to 2 °. Further, when the angle between the horizontal plane and the imaging surface is not in the range of 88 ° to 92 °, it is determined that the condition is not satisfied.

  The setting unit 154 sets an area designated by the user as a predetermined area. The setting unit 154 can set a rectangular region having two vertices designated by the user in the image captured by the imaging unit 13 as a predetermined region. The setting unit 154 may automatically set the area when there is no designation by the user.

  The display control unit 155 displays the image captured by the imaging unit 13 on the display unit 12, and when the determination unit 153 determines that the predetermined condition is not satisfied, the display control unit 155 indicates that the predetermined condition is not satisfied. A message is displayed on the display unit 12. In addition, the display control unit 155 displays a frame 122a indicating a region, a grid 122c indicating a feature plane, and a point 122b indicating a feature point in an overlapping manner on the image captured by the imaging unit 13. In addition, the display control unit 155 is configured so that the mark 122d on the virtual imaging surface of the imaging unit 13 is parallel to the feature surface and the mark 122d when the virtual imaging surface and the feature surface are parallel to each other. The overlapping mark e is further displayed.

  Here, each screen displayed on the display unit 12 by the display control unit 155 will be described with reference to FIGS. 3 to 7. 3 to 7 are diagrams illustrating an example of a screen of the determination apparatus according to the first embodiment. First, as shown in FIG. 3, when the start button 121b is pressed, the display control unit 155 displays a message for explaining the operation.

  As shown in FIG. 4, the display control unit 155 displays a frame 122a indicating the designated area. Further, when the feature point extraction button 121c is pressed, the display control unit 155 displays a point 122b indicating the extracted feature point. At this time, when the number of feature points in the frame 122a is equal to or smaller than a predetermined threshold, the display control unit 155 displays a message “feature point extraction failure, please try again”.

  As shown in FIG. 5, when a part or all of the frame 122a is out of the imaging range, the display control unit 155 displays a message “A feature point is protruding from the screen”.

  As shown in FIG. 6, when the angle between the horizontal plane and the imaging surface is not within a predetermined range, the display control unit 155 displays a message “please level the terminal”. Furthermore, as shown in FIG. 7, when the angle between the feature plane and the imaging plane is not within a predetermined range, the display control unit 155 displays a message “Please keep the terminal vertical”. In the example of FIG. 7, the relationship between the feature surface and the imaging surface is far from parallel, so the distance between the mark 122d and the mark 122e is larger than in the example of FIG.

  When the determination unit 153 determines that the predetermined condition is satisfied, the acquisition unit 156 acquires the marker image captured by the imaging unit 13 together with information specifying a predetermined region. In addition, the acquisition unit 156 further acquires an editing image that includes a predetermined region and has a wider imaging range than the marker image.

  For example, as illustrated in FIG. 8, the acquisition unit 156 acquires an image as an editing image after all of the equipment 20, the frame 122a, and the object 30 are within the imaging range. FIG. 8 is a diagram illustrating an example of a screen of the determination apparatus according to the first embodiment. At this time, the determination apparatus 10 may display a message that prompts the user to pull the imaging unit 13 or may change the focal length of the imaging unit 13 so that the imaging range is expanded.

(Processing of the first embodiment)
With reference to FIG. 9, the determination apparatus 10 sets a region in an image captured based on a user's designation (step S101). And the determination apparatus 10 extracts a feature point from the set area | region (step S102). Here, when the feature points in the area are not sufficient (step S103, No), the determination apparatus 10 displays a message for prompting redesignation of the area (step S104), and returns to the process of step S101.

  On the other hand, when the feature points in the region are sufficient (step S103, Yes), the determination apparatus 10 determines the posture of the imaging unit 13 based on the angle between the feature plane and the horizontal plane and the imaging plane of the imaging unit 13. It is determined whether or not is correct (step S105). Here, when the orientation of the imaging unit 13 is not correct (No at Step S105), the determination apparatus 10 displays a message that prompts the user to correct the orientation of the imaging unit 13 (Step S106), and returns to the process of Step S105.

  On the other hand, when the orientation of the imaging unit 13 is correct (step S105, Yes), it is determined whether or not the region is within the imaging range of the imaging unit 13 (step S107). If the area is not within the imaging range of the imaging unit 13 (step S107, No), the determination apparatus 10 displays a message that the area is not within the imaging range of the imaging unit 13 (step S108). The process returns to S107.

  On the other hand, when the region is within the imaging range of the imaging unit 13 (step S107, Yes), the determination apparatus 10 acquires an image (step S109). At this time, the determination apparatus 10 can acquire the marker image and the editing image.

(Effects of the first embodiment)
In the embodiment, the extraction unit 151 extracts feature points from a predetermined region in the image captured by the imaging unit 13. In addition, the calculation unit 152 calculates an angle between a feature plane and a horizontal plane that are virtual planes generated based on the feature points and the imaging plane of the imaging unit 13. The determination unit 153 determines, based on the feature point and the angle, whether or not a predetermined condition regarding the availability of the image of the predetermined area captured by the imaging unit 13 as the markerless AR marker is satisfied.

  As described above, the determination device 10 can automatically determine whether or not the image being captured can be used as a marker for the markerless AR. For this reason, according to the first embodiment, even a beginner can easily create content using the markerless AR.

  The setting unit 154 sets an area designated by the user as a predetermined area. The determination unit 153 determines that the predetermined condition is not satisfied when at least a part of the predetermined region set by the setting unit 154 is out of the imaging range of the imaging unit 13. This can prevent the extracted feature points from being included in the acquired image.

  The determination unit 153 determines that the predetermined condition is not satisfied when the number of feature points included in the region is equal to or smaller than a predetermined threshold value or when the angle is not within the predetermined range. As a result, it is possible to acquire a highly useful image as a marker image.

  The display control unit 155 displays the image captured by the imaging unit 13 on the display unit 12, and when the determination unit 153 determines that the predetermined condition is not satisfied, the display control unit 155 indicates that the predetermined condition is not satisfied. A message is displayed on the display unit 12. Accordingly, the user can recognize that the image being captured is inappropriate as the marker image.

  The display control unit 155 displays a frame 122a indicating a region, a grid 122c indicating a feature plane, and a point 122b indicating a feature point in an overlapping manner on the image captured by the imaging unit 13. Thus, the user can intuitively grasp the relationship between the imaging surface and the surface where the feature points exist.

  The display control unit 155 is a mark that overlaps the mark 122d when the mark 122d on the virtual imaging surface of the imaging unit 13 is parallel to the feature surface and the virtual imaging surface and the feature surface are parallel to each other. e is further displayed. Thereby, the user can clearly recognize that the imaging surface and the surface on which the feature point exists face each other.

  When the determination unit 153 determines that the predetermined condition is satisfied, the acquisition unit 156 acquires the marker image captured by the imaging unit 13 together with information specifying a predetermined region. As a result, it is possible to acquire a highly useful image as a marker image.

  The acquisition unit 156 further acquires an editing image that is an image including a predetermined area and has a wider imaging range than the marker image. As a result, it is possible to acquire an image for editing in addition to the image for the marker.

  The setting unit 154 sets a rectangular region having two vertices designated by the user in the image photographed by the imaging unit 13 as a predetermined region. Thereby, the user can specify an area | region intuitively.

  Note that the program executed by the determination apparatus 10 of the present embodiment is provided by being incorporated in advance in a ROM or the like. The program executed by the determination apparatus 10 according to the present embodiment may be configured to be recorded and provided on a computer-readable recording medium in an installable format or an executable format file.

  Furthermore, the program executed by the determination apparatus 10 according to the present embodiment may be configured to be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. Further, the program executed by the determination apparatus 10 of the present embodiment may be configured to be provided or distributed via a network such as the Internet.

  The program executed by the determination apparatus 10 of the present embodiment has a module configuration including the above-described units (extraction unit 151, calculation unit 152, determination unit 153, setting unit 154, display control unit 155, and acquisition unit 156). As the actual hardware, the CPU reads the program from the ROM and executes it, so that each unit is loaded onto the main storage device, and the extraction unit 151, the calculation unit 152, the determination unit 153, the setting unit 154, and the display A control unit 155 and an acquisition unit 156 are generated on the main storage device.

  Although embodiments of the present invention have been described, the embodiments have been presented by way of example and are not intended to limit the scope of the invention. The embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. The embodiments and the modifications thereof are included in the scope of the invention and the scope of the invention, and are also included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 10 Determination apparatus 10a User 11 Input part 12 Display part 13 Imaging part 14 Storage part 15 Control part 20 Equipment 30 Object 121a Reset button 121b Start button 121c Feature point extraction button 121d Capture button 122a Frame 122b Point 122c Grid 122d, 122e mark

Claims (11)

An extraction unit that extracts a feature point from a predetermined region in an image captured by an imaging unit that captures an image;
A calculation unit that calculates an angle between a feature plane and a horizontal plane that are virtual planes generated based on the feature point and an imaging plane of the imaging unit;
Based on the feature point and the angle, it is determined whether or not a predetermined condition relating to availability of a markerless AR (Augmented Reality) marker of an image of the predetermined area captured by the imaging unit is satisfied. A determination unit;
The determination apparatus characterized by having. A setting unit that sets an area designated by the user to the predetermined area;
The determination unit determines that the predetermined condition is not satisfied when at least a part of the predetermined region set by the setting unit is out of an imaging range of the imaging unit. The determination apparatus according to claim 1.   The determination unit determines that the predetermined condition is not satisfied when the number of the feature points included in the region is equal to or smaller than a predetermined threshold or when the angle is not within a predetermined range. The determination apparatus according to claim 1, wherein:   An image captured by the imaging unit is displayed on a display unit, and when the determination unit determines that the predetermined condition is not satisfied, a message indicating that the predetermined condition is not satisfied is displayed on the display unit. The determination apparatus according to claim 1, further comprising: a display control unit that displays the display.   5. The display control unit according to claim 4, wherein a frame indicating the region, a grid indicating the feature plane, and a point indicating the feature point are displayed in an overlapping manner on an image captured by the imaging unit. The determination apparatus described.   The display control unit is parallel to the first mark on the virtual imaging surface of the imaging unit and the feature surface, and when the virtual imaging surface and the feature surface are parallel, The determination apparatus according to claim 4, further displaying a second mark that overlaps the first mark.   When the determination unit determines that the predetermined condition is satisfied, the acquisition unit further includes an acquisition unit that acquires the first image captured by the imaging unit together with information specifying the predetermined region. The determination apparatus according to claim 1, wherein:   The determination apparatus according to claim 7, wherein the acquisition unit further acquires a second image that is an image including the predetermined region and has a wider imaging range than the first image.   The said setting part sets the rectangular area | region which makes the two points | pieces designated by the said user the opposite vertex in the image image | photographed by the said imaging part to the said predetermined area | region. Judgment device. A determination method executed by a computer,
An extraction step of extracting feature points from a predetermined region in the image captured by the imaging unit;
A calculation step of calculating an angle between a feature plane and a horizontal plane that are virtual planes generated based on the feature points and an imaging plane of the imaging unit;
Based on the feature point and the angle, it is determined whether or not a predetermined condition relating to availability of a markerless AR (Augmented Reality) marker of an image of the predetermined area captured by the imaging unit is satisfied. A determination process;
The determination method characterized by including. Computer
Extraction means for extracting feature points from a predetermined region in the image captured by the imaging unit;
Calculation means for calculating an angle between a feature plane and a horizontal plane generated based on the feature points and an imaging plane of the imaging unit;
Based on the feature point and the angle, it is determined whether or not a predetermined condition relating to availability of a markerless AR (Augmented Reality) marker of an image of the predetermined area captured by the imaging unit is satisfied. A determination means;
A judgment program characterized by functioning as

Images