JP2019114226A - Program, system, electronic device and method for enabling solid to be recognized - Google Patents

Abstract

To provide a program enabling a solid shape to be recognized from a single image.SOLUTION: A program, which enables a solid made up of predetermined plural components in an actual space to be recognized, is executed by an electronic device mounted with an imaging unit to image the solid. The program causes the electronic device to execute: an image recognition step; a query generation step; a data acquisition step; a component arrangement step; a consistency determination step; a solid recognition step when consistency is determined; and database storage, query regeneration determination and query regeneration steps when the consistency is not determined.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a program, a system, an electronic device, and a method for recognizing a solid, and in particular, a program, a system, an electronic device, and a program for recognizing a solid in a real space composed of a plurality of predetermined parts. On the way.

  There is known a technique for recognizing the three-dimensional shape of a three-dimensional object in real space from a photographed planar image. However, when a solid shape is projected onto a plane, the solid shape can not be directly recognized from a single plane image because solid information on depth and hidden surface is lost. Conventionally, a technique for shape estimation by inductively estimating three-dimensional information by correlating a plurality of images captured from different viewpoints, and deducing three-dimensional information from a priori knowledge of a target for a specific structure Techniques have been developed to estimate shape recognition.

  For example, Non-Patent Document 1 discloses a technique for shape recognition by associating a plurality of images using a technique called Structure from Motion (SfM). Further, for example, Non-Patent Document 2 discloses a technique of shape recognition using a single image as an input and assuming a specific object such as a human body or an expression. The technology focuses on the fact that human facial expressions are globally similar and there are many local differences, and by transforming an existing 3D model, a 3D model corresponding to an input image is constructed. is there.

Marc Pollefeys, Luc Van Gool, Maarten Vergauwen, Frank Verbiest, Kurt Cornelis, Jan Tops, Reinhard Koch. 2004. Visual Modeling with a Hand-Held Camera. International Journal of Computer Vision. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. 2011. 3D face reconstruction from a single image using a single reference face shape.

  However, in the technique disclosed in Non-Patent Document 1, it is necessary for the user to perform photographing a plurality of times or to perform photographing using a plurality of cameras, which complicates the user operation. Further, the technology disclosed in Non-Patent Document 2 can not be applied to recognition of a structure that a user can freely create, since it is premised that the structure is known. As described above, a three-dimensional shape recognition technique from a single plane image for a free-form structure has not yet been established.

  The present invention has been made to solve such problems, and its main object is to provide a program and the like capable of recognizing a three-dimensional shape from a single image.

  In order to achieve the above object, a program as an aspect of the present invention is a program for recognizing a solid in a real space composed of a plurality of predetermined parts, the program comprising the solid This method is executed in an electronic device provided with a photographing device for photographing, and the electronic device extracts features capable of identifying each component from the image photographed by the photographing device, and the component is calculated based on the characteristic An image recognition step of acquiring each ID of the part and position data of the part, a query generation step of generating a query including each ID of the part acquired based on the feature, and a query is generated, Component data that associates and stores the ID of each component, the three-dimensional model of each component, and the constraint data indicating the bonding constraint between each component using a query From the data acquisition step of acquiring three-dimensional models and constraint data of each of the parts associated with each of the IDs included in the query, and position data, three-dimensional models, and restriction data of each of the acquired parts. Alignment based on the component arrangement step of arranging each component in the virtual space corresponding to the real space based on the positional relationship of the arranged components, the alignment of the positions of the arranged components is determined Data determination step, and if it is determined that the consistency is present, if it is determined that the 3D recognition step to recognize a solid based on the position of each of the arranged parts, and if the consistency is not present, the data acquisition A database storage unit for storing, in a history database, query storage data including at least the ID of each component among the queries used in the step A query regeneration determination step of deciding whether to regenerate a query based on the query storage data stored in the history database and the data stored in the component database, and the query is determined to be regenerated And a query regenerating step of generating a query including data different from the query storage data stored in the history database.

  In the present invention, preferably, the program causes the electronic device to execute a database initialization step of deleting data for storing query stored in the history database before the image recognition step.

  In the present invention, preferably, the component placement step places each component in the virtual space using the acquired position data of each component and a three-dimensional model, and each of the adjacent components in the virtual space In the above, the position of each part is adjusted and arranged so as to satisfy the joining constraint.

  In the present invention, preferably, the component placement step is performed based on whether or not the length between three-dimensional models of two components of the components disposed in the virtual space is within a predetermined value. It is determined whether or not two parts are adjacent.

  In the present invention, preferably, the position is a position with six degrees of freedom.

  In the present invention, preferably, the feature that can identify each part is an AR marker that each part has on the part surface.

  In the present invention, preferably, when the query is generated, the data acquisition step uses the query to generate a three-dimensional model of each of the components associated with each of the IDs included in the query from the component database and Constraint data is acquired by being associated with the ID and added to the query.

  In the present invention, preferably, the consistency determination step determines the consistency of the positions of the arranged parts based on the overlapping degree of the area occupied by the three-dimensional model of the arranged parts.

  In the present invention, preferably, the image recognition step recognizes a base portion that forms the base portion of the solid from the image captured by the imaging device, and the component placement step includes the base portion in the virtual space. The arrangement is performed on the upper side, and the consistency determination step determines whether or not the arranged parts are arranged according to gravity from the positional relationship between the positions of the arranged parts and the position of the base portion. And determine consistency based further on the determination.

  Furthermore, in the present invention, preferably, the consistency determination step is arranged at the position of the virtual space corresponding to the position of the image pickup device of the image photographed by the image pickup device and each component arranged. Consistency is determined based on the degree of coincidence with the image viewed from the virtual camera.

  In the present invention, preferably, the database storage step further stores the position of each of the arranged parts in the history database in association with the ID of each of the parts included in the query storage data.

  In the present invention, preferably, the query regeneration determination step is performed based on whether or not the query storage data stored in the history database includes a predetermined combination of component IDs stored in the component database. Decide whether to regenerate the query.

  In the present invention, preferably, the program causes the electronic device to further execute an error output step of outputting an error if it is determined not to regenerate the query.

  In the present invention, preferably, the database storage step further stores, in the history database, an evaluation value of the consistency of the position of each of the arranged parts in association with the ID of each of the parts included in the query storage data. Do.

  In the present invention, preferably, when it is determined that the program does not regenerate a query in the electronic device, the program is three-dimensional based on the positions of the arranged parts having the highest consistency with the evaluation value. To perform a second three-dimensional recognition step of recognizing

  In the present invention, preferably, in the query regenerating step, when a length between three-dimensional models of two parts of the parts arranged in the virtual space is a predetermined value or more, Generate a query with the corresponding part ID added.

  Further, in order to achieve the above object, a three-dimensional recognition system as one aspect of the present invention is a three-dimensional recognition system for recognizing a three-dimensional object in a real space composed of a plurality of predetermined parts, An electronic apparatus comprising an imaging device for photographing the solid, a component database for storing ID of each component, a three-dimensional model of each component, and constraint data indicating joining constraints among the components, and the component Among the queries input to the database, a history database storing query storage data including at least the ID of each component, and features capable of identifying each component from the image captured by the imaging device are extracted, respectively, An image recognition unit for acquiring each ID of the part and position data of the part based on the information, and that of the part acquired based on the feature A query generation unit that generates a query including the ID of the query, and, when the query is generated, a three-dimensional model of each of the components associated with each of the IDs included in the query from the component database using the query A data acquisition unit that acquires constraint data; a component arrangement unit that arranges each component in a virtual space corresponding to the real space based on position data, a three-dimensional model, and constraint data of each of the acquired components; A consistency determination unit that determines the alignment of the positions of the arranged parts based on the positional relationship of the arranged parts, and when it is determined that the alignment is present, the arranged parts A solid recognition unit that recognizes a solid based on the position of the object, and the ID of at least each of the parts of the query used by the data acquisition unit when it is determined that the On the basis of the database storage unit for storing the query storage data in the history database, the query storage data stored in the history database, and the data stored in the component database. The apparatus is characterized by comprising: a query regeneration determination unit; and a query regeneration unit that generates a query including data different from the query storage data stored in the history database, when it is determined to regenerate the query. .

  Further, in order to achieve the above object, an electronic device according to one aspect of the present invention is an electronic device for recognizing a solid in a real space composed of a plurality of predetermined parts, wherein the solid is A part database that associates and stores an imaging device for capturing an ID of each part, an ID of each part, a three-dimensional model of each part, and constraint data indicating joining constraints between each part, and a query input to the part database And a history database for storing query storage data including at least the ID of each of the parts, and a feature capable of identifying each of the parts from the image captured by the imaging device, and extracting each of the parts based on the feature Generating a query including an ID and an image recognition unit for acquiring position data of the part, and each ID of the part acquired based on the feature A query generation unit; and a data acquisition unit that acquires, using the query, a three-dimensional model of each of the components associated with each of the IDs included in the query and constraint data when the query is generated. A component placement unit for placing each component in a virtual space corresponding to the real space based on the acquired position data of each component, the three-dimensional model, and the constraint data, and the positional relationship between the disposed components A consistency determination unit that determines the consistency of the position of each of the arranged parts based on the above, and when it is determined that there is the consistency, a solid is recognized based on the position of each of the arranged parts If it is determined that there is no consistency between the three-dimensional recognition unit, the query storage data including at least the ID of each part of the queries used by the data acquisition unit is the history A query storage determination unit configured to determine whether to regenerate a query based on a database storage unit stored in a database, query storage data stored by the history database, and data stored by the component database; Are determined to be regenerated, and a query regenerating unit that generates a query including data different from the query storage data stored in the history database.

  Further, in order to achieve the above object, a method as an aspect of the present invention is an imaging method for photographing a solid for recognizing a solid in a real space composed of a plurality of predetermined parts. A method implemented in an electronic device comprising the device, wherein a feature capable of identifying each component is extracted from the image captured by the imaging device, and the ID of the component and the position of the component based on the feature. In the image recognition step for acquiring data, the query generation step for generating a query including the ID of each of the parts acquired based on the feature, and when the query is generated, using the query, the ID of each part , A three-dimensional model of each part, and a parts database that associates and stores constraint data indicating joining constraints between the parts, each of the IDs included in the query Data acquisition step for acquiring three-dimensional model and restriction data of each attached part, and virtual space corresponding to the real space based on position data, three-dimensional model, and restriction data of each acquired part A consistency determining step of determining the alignment of the positions of the arranged parts based on the part arranging step of arranging the parts and the positional relationship of the arranged parts, the judgment that the alignment is present If it is determined that there is no matching, it is determined that there is no matching, and if at least one of the IDs of each of the components of the query is used, Database storing step of storing data for storing queries including the above in a history database, and a query stored in the history database A query regeneration determination step of determining whether to regenerate a query based on data for storage and data stored in the parts database, and a query stored in the history database when it is determined to regenerate the query And D. a query regenerating step of generating a query including data different from the storage data.

  According to the present invention, a three-dimensional shape can be recognized from a single image.

BRIEF DESCRIPTION OF THE DRAWINGS It is a whole block diagram of the three-dimensional recognition system by one Embodiment of this invention. It is a block diagram showing the hardware constitutions of the electronic device by one embodiment of the present invention. It is a block diagram showing the hardware constitutions of the server by one embodiment of the present invention. FIG. 6 shows an example of a functional block diagram of a server according to an embodiment of the present invention. FIG. 1 shows an example of a functional block diagram of an electronic device according to an embodiment of the present invention. It is a figure which shows an example of the imaging | photography image by an imaging device. It is a figure which shows three components which the image recognition part recognized from the picked-up image shown in FIG. It is a figure which shows a mode that the group of two components which a components arrangement | positioning part adjoins is specified from the three components which the image recognition part shown in FIG. 7 recognized, respectively, and it moves so that a joining restrictions condition may be satisfy | filled. It is a figure which shows the structure which the three-dimensional recognition part recognized by the components which the components arrangement | positioning part shown in FIG. 8 decided arrangement | positioning. It is a flowchart which shows the information processing of the three-dimensional recognition system by one Embodiment of this invention.

  Hereinafter, a three-dimensional recognition system 1 according to an embodiment of the present invention will be described with reference to the drawings. In the present specification, for convenience of description, the description may be omitted in detail. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially the same configurations may be omitted.

  The three-dimensional recognition system 1 according to the embodiment of the present invention is a system for recognizing a three-dimensional object in real space created by combining a plurality of predetermined parts 5 whose shapes and sizes are known. In the present specification, a solid in real space created by combining a plurality of parts 5 may be particularly referred to as a structure. One of the technical features of the embodiment of the present invention is that the three-dimensional recognition system 1 recognizes the structure created as described above from a single plane image. It is understood that even if the shape and size of the individual parts 5 are known, the structure created by freely combining the parts 5 is not known.

  It should be noted that the part 5 in the real space in the present specification does not substantially change in volume or shape when the other part 5 contacts or when the other part 5 is placed on the part 5 It assumes a rigid body. Also, in the present specification, the contact of two parts 5 to form one structure is referred to as joining of two parts 5. For example, joining two parts 5 means that one part 5 and another part 5 are in contact, that one part 5 and another part 5 are fitted and fixed, and one part 5 is It shows that it can be placed on another component 5 or the like.

  FIG. 1 is an overall configuration diagram of a three-dimensional recognition system 1 according to an embodiment of the present invention. As shown in FIG. 1, the three-dimensional recognition system 1 includes an electronic device 3 and a server 4. The electronic device 3 and the server 4 are connected to a network 2 such as the Internet and can communicate with each other.

  FIG. 2 is a block diagram showing the hardware configuration of the electronic device 3 according to an embodiment of the present invention. The electronic device 3 includes a processor 11, a display device 12, an input device 13, an imaging device 14, a storage device 15, and a communication device 16. Each of these components is connected by a bus 16. An interface is interposed between the bus 17 and each component device as necessary. In the present embodiment, the electronic device 3 is a smartphone. However, as long as the electronic device 3 has the above configuration, it can be a portable electronic terminal such as a computer including a tablet computer and a contact input device such as a touch pad.

  The processor 11 controls the overall operation of the electronic device 3 and is, for example, a CPU. Note that an electronic circuit such as an MPU may be used as the processor 11. The processor 11 executes various processes by reading and executing programs and data stored in the storage device 14. In one example, processor 11 is comprised of a plurality of processors.

  The display device (display) 12 displays the application screen, an image photographed by the photographing device 14 and the like to the user of the electronic device 3 under the control of the processor 11. A liquid crystal display is preferable, but a display using an organic EL, a plasma display, or the like may be used.

The input device 13 is a user interface that receives an input from the user to the electronic device 3 and is, for example, a touch panel, a touch pad, a keyboard, or a mouse.
In the present embodiment, since the electronic device 3 is a smartphone, the electronic device 3 includes a touch panel as the input device 13, the touch panel also functions as the display device 12, and the display device 12 and the input device 13 are integrated. . However, the display device 12 and the input device 13 may be separate forms disposed at different positions.

  The imaging device (imaging device) 14 captures (captures) a still image or a moving image in the real space, and stores the captured image or the moving image data in the storage unit 15. The imaging device 14 is a camera configured of, for example, an image sensor.

  The storage device 15 is a storage device provided in a general smartphone including a RAM which is a volatile memory and a ROM which is a non-volatile memory. The storage device 15 can also include an external memory.

  In one example, the storage device 15 includes a main storage device and an auxiliary storage device. The main storage device is a volatile storage medium capable of high-speed reading and writing of information, and is used as a storage area and a work area when the processor 11 processes information. The auxiliary storage device stores various programs and data used by the processor 11 when executing each program. The auxiliary storage device is, for example, a hard disk device, but may be any non-volatile storage or non-volatile memory as long as it can store information, or may be removable. The auxiliary storage device stores, for example, an operating system (OS), middleware, an application program, various data that can be referred to with the execution of these programs, and the like.

  The communication device 16 exchanges data with another computer such as the server 4 via the network 2. For example, the communication device 16 performs wireless communication such as mobile communication and wireless LAN, and connects to the network 2. In one example, the communication device 16 downloads the program from the server 4 and stores the program in the storage device 16. However, the communication device 16 may perform wired communication using an Ethernet (registered trademark) cable or the like.

  FIG. 3 is a block diagram showing the hardware configuration of the server 4 according to an embodiment of the present invention. The server 4 includes a processor 21, a display device 22, an input device 23, a storage device 24, and a communication device 25. Each of these components is connected by a bus 26. An interface is interposed between the bus 26 and each component device as necessary.

  The processor 21 controls the overall operation of the server 4 and is, for example, a CPU. Note that an electronic circuit such as an MPU may be used as the processor 11. The processor 11 executes various processes by reading and executing programs and data stored in the storage device 14. In one example, processor 21 comprises multiple processors.

  The display device (display) 22 displays an application screen or the like to the user of the server 4 according to the control of the processor 21. The input device 23 receives an input from the user to the server 4 and is, for example, a touch panel, a touch pad, a keyboard, or a mouse.

  The storage device 24 includes a main storage device and an auxiliary storage device. The main storage device is, for example, a semiconductor memory such as a RAM. The RAM is a volatile storage medium capable of high-speed reading and writing of information, and is used as a storage area and a work area when the processor 21 processes information. The main storage device may include a ROM, which is a read only non-volatile storage medium. In this case, the ROM stores a program such as firmware. The auxiliary storage device stores various programs and data used by the processor 21 when executing each program. The auxiliary storage device is, for example, a hard disk device, but may be any non-volatile storage or non-volatile memory as long as it can store information, or may be removable. The auxiliary storage device stores, for example, an operating system (OS), middleware, an application program, various data that can be referred to with the execution of these programs, and the like.

  In one preferable example, the storage unit 24 stores data (for example, a table) and programs for various databases. Various databases are realized by the operation of the processor 21 and the like. The server 4 may have a database server function, may include one or more database servers, or may include other servers.

  The communication device 25 is a device for exchanging data with another computer such as the electronic device 3 via the network 2. For example, the communication device 25 performs wired communication using an Ethernet (registered trademark) cable or the like, mobile communication, and wireless communication such as wireless LAN, and connects to the network 4.

  FIG. 4 shows an example of a functional block diagram of the server 4 according to an embodiment of the present invention. The server 4 includes a parts database 41 and a history database 42. In the present embodiment, these functions are realized by the processor 21 executing a program. As described above, since various functions are realized by program reading, a part of one database may be included in another database.

  As described above, the three-dimensional recognition system 1 recognizes a structure in the real space composed of a plurality of predetermined parts 5. The part database 41 associates and stores a part ID of each of the parts 5, three-dimensional model data (three-dimensional model) of the part 5, and constraint data indicating a joint constraint between the parts 5. The electronic device 3 acquires data from the component database 41 using a query.

  The part ID is data indicating unique identification information for identifying the part 5. The parts 5 stored in the parts database 41 are each associated with a part ID. The structure is created by part or all of the parts 5 stored in the parts database 41. When there are a plurality of parts 5 having the same shape and size, the parts 5 have the same part ID. In this case, the component database 41 further stores the number of components 5 in association with the component ID. However, all parts 5 may be configured to have different part IDs, and the parts database 41 may not include the number of parts 5.

  The three-dimensional model is data that defines the shape and size of the component 5 and is data represented by, for example, a wire frame model or a polygon mesh.

  Constraint data is alignment metadata for defining a bonding constraint between parts 5 stored in the parts database 41. The joining constraint is a geometric constraint imposed when joining the parts 5 and determines the relative positional relationship between the two parts 5.

  The alignment metadata is data that defines at least one constraint out of six degrees of freedom (6 DoF) in the relative positional relationship between the two parts 5. 6DoF is a degree of freedom consisting of coordinates and rotation directions with respect to the position and posture of a solid in a three-dimensional space. The parts database 41 associates and stores alignment metadata for defining constraints with each of the other parts 5 with the part ID of one part 5.

  The alignment metadata of one part 5, that is, the alignment metadata associated with the part ID of one part 5, defines constraints with each of all the parts 5 that may be joined with one part 5. Contains data. In one example, in the case where the part database 41 includes only one part 5, the alignment metadata associated with the part ID of one part 5 includes all other parts 5 other than the one part 5. And data that defines the constraints. In one example, when the part database 41 includes a plurality of parts 5, the alignment metadata associated with the part ID of the part 5 is a restriction with each of all the parts 5 including the part 5. Contains data that defines

  The parts database 41 stores the three-dimensional model of the parts 5 and the alignment metadata using a coordinate system common to the parts 5. Preferably, the three-dimensional model and alignment metadata for each part 5 stored in the parts database 41 are represented by a three-axis coordinate system with the center of each part 5 as the origin.

（式１）

（式２）

（式３）
式１から式３において、

は、一の部品５の接合基準点の座標を示す。

は、一の部品５の接合基準点と他の部品５の接合基準点とのｘ軸方向の距離の最小値と最大値を示す。

は、一の部品５の接合基準点と他の部品５の接合基準点とのｙ軸方向の距離の最小値と最大値を示す。

は、一の部品５の接合基準点と他の部品５の接合基準点とのｘ軸方向の距離の最小値と最大値を示す。

は、一の部品５の接合基準点と他の部品５の接合基準点とのｘ軸を中心とした回転角度差（ｒｏｌｌの回転角度差）の最小値と最大値を示す。

は、一の部品５の接合基準点と他の部品５の接合基準点とのｙ軸を中心とした回転角度差（ｐｉｔｃｈの回転角度差）の最小値と最大値を示す。

は、一の部品５の接合基準点と他の部品５の接合基準点とのｚ軸を中心とした回転角度差（ｙａｗの回転角度差）の最小値と最大値を示す。 In one preferred example, when any point on the part surface is defined as the bonding reference point 6 in all parts 5, the alignment metadata of one part 5 is the same as the other with respect to the bonding reference point 6 of one part 5. It is determined by the conditions of the distance and the angle of the bonding reference point 6 of the component 5 of FIG. Focusing on a set of parts 5, the alignment metadata (alignment-data) of one part 5 is an array having positional constraints (or orientational constraints) and orientational constraints (orientational constraints). It can be described as Formula 1 to Formula 3 shown below.

(Formula 1)

(Formula 2)

(Equation 3)
In formulas 1 to 3,

These show the coordinates of the junction reference point of one component 5.

These show the minimum value and the maximum value of the distance of the x-axis direction of the junction reference point of one component 5 and the junction reference point of the other component 5.

These show the minimum value and the maximum value of the distance of the y-axis direction of the junction reference point of one component 5 and the junction reference point of the other component 5.

These show the minimum value and the maximum value of the distance of the x-axis direction of the junction reference point of one component 5 and the junction reference point of the other component 5.

These show the minimum value and the maximum value of the rotational angle difference (rotational angle difference of a roll) centering on the x-axis of the joint reference point of one component 5 and the joint reference point of the other component 5.

These show the minimum value and the maximum value of the rotational angle difference (rotational angle difference of pitch) centering on the y-axis of the joint reference point of one part 5 and the joint reference point of the other part 5.

These show the minimum value and the maximum value of the rotational angle difference (rotational angle difference of yaw) centering on the z-axis of the joint reference point of one part 5 and the joint reference point of the other part 5.

  The alignment metadata shown in Equation 1 to Equation 3 is data that defines the constraint between one component 5 and another one component 5. The part database 41 stores alignment metadata defining constraints with each of the parts 5 that may be joined with one of the parts 5 in association with the part ID of the one part 5. When one component 5 has a plurality of joint surfaces, one component 5 can have a plurality of joint reference points. In this case, with regard to the data defining the constraint between one part 5 and the other one part 5, the alignment metadata may have data shown in equations 1 to 3 for each junction reference point, and Elements of posture constraints can be treated as data subordinate to the junction reference point. Hereinafter, in the present embodiment, it is assumed that constraint data of one component 5 has alignment metadata for each component 5 that may be joined to the one component 5. Further, the alignment metadata has elements of position constraint and attitude constraint as the joint reference point 6 and data subordinate to the joint reference point 6.

（式４）

（式５）

（式６）

（式７）

（式８）

（式９） In one example, in the case where two parts 5 (P ₁ and P ₂ ) each having one joint reference point are joined, the following equation is established if the alignment data of each part 5 is represented by a subscript: .

(Equation 4)

(Equation 5)

(Equation 6)

(Equation 7)

(Equation 8)

(Equation 9)

  Since the direction of rotation of the solid may be different depending on the order of selection of the axes, for example, in one application, the order is defined as one. In one example, the minimum value and the maximum value of at least one distance between the joint reference point of one part and the joint reference point of any other part in the x-axis, y-axis and z-axis directions [0 , ∞], and at least one minimum value and maximum value of roll, pitch, and yaw between the joint reference point of one part and the joint reference point of any other part are [-180, 180]. This leaves some freedom in the constraints. In another example, the alignment metadata is configured not to define any one of the 6 DoF constraints. In this case, the parts database 41 may not store constraint data.

  The history database 42 stores query storage data including at least the ID of each component among the queries input from the electronic device 3 to the component database 41. The query input to the parts database 41 is, for example, a query used when the electronic device 3 searches and acquires data from the parts database 41.

  FIG. 5 shows an example of a functional block diagram of the electronic device 3 according to an embodiment of the present invention. The electronic device 3 includes an image recognition unit 31, a query generation unit 32, a data acquisition unit 33, a component placement unit 34, a consistency determination unit 35, a solid recognition unit 36, a database storage unit 37, and query reproduction. And a query re-generation unit 39. In the present embodiment, these functions are realized by the processor 11 executing a program. As described above, since various functions are realized by program reading, some or all of one part (function) may be included in another part. However, these functions may also be realized by hardware by configuring an electronic circuit or the like for realizing a part or all of the functions.

  First, the component 5 used in the three-dimensional recognition system 1 will be described. The plurality of parts 5 have markers that can identify each part 5 on the part surface. In one preferable example, the marker is an AR marker, and is constituted of, for example, a rectangular black frame and an asymmetric dot pattern drawn inside the black frame. Thus, the marker has a shape capable of acquiring the position and orientation of the marker from the imaging device 14 from the image in which the marker is photographed, and has a feature pattern capable of identifying each component 5. However, the plurality of components 5 may have features that can identify each component 5 on the component surface other than a marker that can identify each component 5. For example, the plurality of parts 5 have only feature patterns that can identify each part 5, and the feature patterns can acquire the position and orientation of the feature patterns from the imaging device 14 from the captured image. , And each component 5 may be identified.

  The image recognition unit 31 extracts markers that can identify each component 5 from the image captured by the imaging device 14. The image recognition unit 31 acquires the component ID of the component 5 from the extracted marker, and acquires position data of each component 5 from the shape and size of the marker in the captured image. Thus, the image recognition unit 31 recognizes each component 5 in the captured image. In one example, the position data of each component 5 acquired by the image recognition unit 31 is the position of the reference point of recognition of the marker represented by the camera coordinate system, and indicates the relative position from the photographing device 14 . In one example, the position data is a 6 DoF (6 degrees of freedom) position. In another example, position data is expressed by quaternion or the like. The image recognition unit 31 can have the function of a known image recognition technology, and the position data of each component 5 acquired by the image recognition unit 31 may be a rough position.

  The part database 41 stores the shape, size, and characteristic pattern of the marker on the surface of the part 5 associated with the part ID and the position of the marker on the part 5 in association with the part ID. The position of the marker in the component 5 is, for example, the position of the reference point of recognition of the marker recognized by the image recognition unit 31, and is represented by a three-axis coordinate system whose origin is the center of the component 5. Further, the parts database 41 stores the position of the bonding reference point 6 of the part 5 associated with the part ID in association with the part ID. The position of the junction reference point 6 is stored in the same manner as the position of the part 5 of the marker, and is represented by, for example, a three-axis coordinate system whose origin is the center of the part 5.

  FIG. 6 is a view showing an example of a photographed image by the photographing device 14. In the photographed image, a structure composed of three parts 5 (5a, 5b, 5c) appears. The image recognition unit 31 extracts three markers from the captured image shown in FIG. 6 and acquires component IDs associated with the three components 5 from the extracted markers and position data of the components 5, The three parts 5 are recognized respectively. FIG. 7 is a diagram showing the three components 5a, 5b, 5c recognized by the image recognition unit 31 from the photographed image shown in FIG. 6, and the bonding reference points 6 (6a, 6b1, 6b2, 6c) of the three components 5 are shown. ). It can be confirmed that the part 5b has two joint reference points 6b1 and 6b2. For example, when the marker of the central part is not shown in the captured image, the image recognition unit 31 recognizes only the parts 5 at both ends.

  The query generation unit 32 generates a query including the component ID acquired by the image recognition unit 31. When the query is generated, the data acquisition unit 33 acquires, using the generated query, the three-dimensional model and constraint data of each of the parts 5 associated with each of the part IDs included in the query from the parts database 41 Do.

  The query includes a part ID and data subordinate to the part ID, and is defined, for example, as an array having each data as an element. The data subordinate to the part ID includes position data of the part 5 associated with the part ID. The position data of the component 5 is acquired by the image recognition unit 31. Further, data subordinate to the part ID includes a three-dimensional model and constraint data.

  The query used when the data acquisition unit 33 acquires data may include the component ID acquired by the image recognition unit 31 and the position data of the component 5 associated with the component ID. In one example, the data acquisition unit 33 sets, in the query used in the search of the component database 41, the three-dimensional model of each of the components 5 associated with the component ID included in the query and the constraint data to the component ID. Match and add to the query respectively. Thus, the data acquisition unit 33 acquires the three-dimensional model of the component 5 and the constraint data for each component ID.

  The component placement unit 34 associates the real space with the real space based on the position data acquired by the image recognition unit 31 and the three-dimensional model and constraint data acquired by the data acquisition unit 33 for each component ID included in the query. Each part 5 is arranged in the virtual space. First, the component placement unit 34 places each component 5 in the virtual space in the virtual space using the three-dimensional model and the position data for each component ID included in the query.

  Next, the component placement unit 34 identifies a set of two adjacent components 5 among the disposed components 5 by determining whether each of the disposed components 5 is adjacent. In one example, the component placement unit 34 determines whether the length between the bonding reference points 6 of the two components 5 placed first is within a predetermined value, and if it is within the predetermined value, 2 It is determined that the two parts 5 are adjacent. The component placement unit 34 can determine that the plurality of components 5 are adjacent to one component 5.

  Next, the component placement unit 34 selects a set of components 5 in a predetermined order from the pair of two components 5 identified as being adjacent. The component placement unit 34 moves or rotates any one of the two components 5 so as to satisfy the bonding restriction condition in each of the two sets of the components 5 determined to be adjacent (hereinafter referred to as “ Let the movement etc.))), and the arrangement is decided. Thus, the component placement unit 34 adjusts and places each component 5. The component placement unit 34 does not move the component 5 or the like to determine the placement when the joint constraint condition is satisfied. The component placement unit 34 does not move the component 5 determined not to be adjacent to any component 5 so as to satisfy the bonding restriction condition.

  In one example, the component placement unit 34 selects the set of two components 5 in order from the combination of two components 5 including the component 5 with high priority. In this case, the component database 41 further stores priority information on the priority of the component 5 associated with the component ID in association with the component ID. The priority of the component 5 is predetermined by the system administrator, and for example, the priority of the component 5 that can be important in constructing the structure is set to be higher. When the query is generated, the data acquiring unit 33 further acquires, from the component database 41, the priority information of each of the components 5 associated with each of the component IDs included in the query, using the generated query. In another example, the image recognition unit 31 recognizes the component ID and calculates the accuracy of the recognition, and the component placement unit 34 sequentially executes the two sets of two components 5 including the component 5 with high accuracy. Select a set of five parts.

  In one example, in the case where one of the two parts 5 needs to be moved or the like, the part arranging unit 34 does not determine the arrangement of one of the parts 5 if it has already been determined. The component 5 is moved or the like so as to satisfy the bonding restriction condition. In another example, in the case where it is necessary to move one of the two parts 5 or the like, the component placement unit 34 may place the component 5 with low priority if the placement of both components 5 has already been determined. The part 5 with low recognition accuracy of the image recognition unit 31 is moved or the like so as to satisfy the bonding restriction condition.

  FIG. 8 identifies the set of two parts 5 adjacent to each other from the three parts 5 recognized by the image recognition section 31 shown in FIG. FIG.

  The consistency determination unit 35 determines the consistency of the positions of the arranged parts 5 based on the positional relationship of the parts 5 arranged by the parts arrangement unit 34.

  In one example, the consistency determination unit 35 determines the consistency of the position of each arranged component 5 based on the overlapping degree of the area occupied by the three-dimensional model of each arranged component 5. Specifically, the consistency determination unit 35 determines that there is consistency when the overlap of the area occupied by the three-dimensional model of each of the arranged parts 5 is (a) within a predetermined volume, (b) If it exceeds the predetermined volume, it is determined that there is no consistency. The predetermined volume described above may be zero or may be determined for each three-dimensional model of each part 5.

  In one example, the image recognition unit 31 recognizes a base portion that forms a base portion on which a structure is placed from image data captured by the imaging device 14. For example, when the structure is placed on a desk, the surface of the placed desk is the base. The component placement unit 34 places the base unit in the virtual space, and determines whether or not each of the placed components 5 is adjacent to the base unit. When the component placement unit 34 moves the base unit and the component 5 so as to satisfy the bonding restriction conditions, it is preferable to move the component 5 or the like to determine the placement. From the positional relationship between the position of each of the arranged components 5 and the position of the base portion, the consistency determination unit 35 determines whether or not each of the arranged components 5 is arranged according to gravity. Specifically, the consistency determination unit 35 determines that there is consistency in the case of the arrangement in accordance with gravity, and determines that there is no consistency in the case of not the arrangement in accordance with gravity.

  For example, when the image recognition unit 31 recognizes only the components 5 at both ends in FIG. 6, the query generated by the query generation unit 32 does not include the component ID of the component 5b, and the component placement unit 34 puts the components 5a and 5c in virtual space. Deploy. The component placement unit 34 recognizes the direction of gravity from the positional relationship between the base portion and the component 5c, and because the component 5a is disposed with a gap from the component 5c, it is determined that the component 5a is not placed according to gravity. And determine that there is no consistency.

  In one example, the consistency determination unit 35 determines the matching degree between the image captured by the imaging device 14 and the image viewed from the virtual camera of each component 5 disposed in the virtual space by the component placement unit 34. , Determine consistency. Specifically, the consistency determination unit 35 determines that there is consistency when the degree of coincidence is equal to or greater than a predetermined value determined in advance, and determines that there is no consistency when the degree of coincidence is less than the predetermined value. Do. Here, the virtual camera is disposed at the position of the virtual space corresponding to the position of the photographing device 14.

  In one example, when the consistency determination unit 35 determines that there is no consistency, it calculates an evaluation value of the consistency of the position of each of the parts 5 arranged by the parts arrangement unit 34. For example, in the case where the evaluation value is determined based on the overlapping degree of the area occupied by the three-dimensional model of each part 5 in which the consistency determining unit 35 is arranged, the evaluation value is calculated according to the excess degree to the predetermined volume of the overlapping area. Be done. Further, for example, when the consistency determination unit 35 determines based on the degree of coincidence of image data, the evaluation value is calculated according to the degree of coincidence.

  The determination of consistency performed by the consistency determination unit 35 as described above is for the entire part 5 disposed in the virtual space, but the consistency determination unit 35 determines the consistency for each part 5 can do. In one example, when determining the consistency of the position of each component 5 based on the overlapping degree of the area occupied by the three-dimensional model of each component 5 in which the consistency determination unit 35 is arranged, the component having the higher priority It is determined that 5 is consistent, and the component 5 with lower priority is not consistent.

  The three-dimensional recognition unit 36 recognizes a structure based on the position of each of the components 5 arranged by the component arrangement unit 34 when the consistency judgment unit 35 determines that the consistency is present. For example, the three-dimensional recognition unit 36 recognizes a structure by the three-dimensional model and position data of each part 5 arranged in the virtual space by the part arrangement unit 34, and recognizes a three-dimensional model of the whole structure. Preferably, the three-dimensional recognition unit 36 reconstructs a three-dimensional model of the entire recognized structure on a virtual space.

  FIG. 9 is a view showing a structure recognized by the three-dimensional recognition unit 36 by the component 5 whose arrangement has been determined by the component arrangement unit 34 shown in FIG. 8. 6 to 9, the priority is higher in the order of the parts 5c, 5b, 5a, and the part placement unit 34 moves the part 5b in the set of parts 5b and 5c, and the part 5a in the set of parts 5b and 5a. It can confirm that it was moved.

  If the consistency determination unit 35 determines that there is no consistency, the database storage unit 37 stores the query storage data in the history database 42. The query storage data includes the component ID of each component 5 and the bonding reference point 6 used by the component placement unit 34 among the data included in the query used when the data acquisition unit 33 acquires data from the component database 41. It is data. The bonding reference point 6 used by the component placement unit 34 is the bonding reference point 6 used by the component placement unit 34 for the determination of the bonding constraint condition among the bonding reference points 6 included in the constraint data associated with the component ID. . The information on the joint reference point 6 is included in the alignment data stored for each adjacent part 5, so the query storage data can specify the adjacency relation of the parts 5 arranged by the parts arranging unit 34. It is a thing. However, among the data included in the query used when the data acquisition unit 33 acquires data from the parts database 41, the data for query storage may be data including at least the part ID of each of the parts 5, and the above embodiment It is not limited to.

  In one example, the database storage unit 37 further stores the evaluation value of the consistency of the position of each component 5 arranged by the component arrangement unit 34 in the history database 42 in association with each of the component IDs included in the query storage data. Do. In this case, when the consistency determination unit 35 determines that there is no consistency, the evaluation value of the consistency is calculated. With such a configuration, when the consistency determination unit 35 determines that there is no consistency in all cases, it is possible to recognize the structure using the most consistent query. .

  In one example, the database storage unit 37 further stores the position of each component 5 arranged by the component arrangement unit 34 in the history database 42 in association with each of the component IDs included in the query storage data. With such a configuration, when the consistency determination unit 35 determines that there is no consistency in all cases, a structure is obtained using the position of each part 5 at the time of the most consistent arrangement. It is possible to recognize

  In one example, the query storage data is data of the query itself, and the database storage unit 37 stores the query used when the data acquisition unit 33 acquires data from the component database 41 in the history database 42 as it is.

When the consistency determination unit 35 determines that there is no consistency, the query regeneration determination unit 38 regenerates a query based on the query storage data stored in the history database 42 and the data stored in the component database 41. It is determined whether to do. Specifically, the query regeneration determination unit 38 determines whether the query storage data stored in the history database 42 includes a predetermined one of a combination of part or all of the component IDs stored in the component database 41. Determine if The query regeneration determination unit 38 determines that the query is to be regenerated if the query storage data stored in the history database 42 does not include the predetermined one, and if the query storage data includes the predetermined one, the query is generated. Decide not to regenerate. The combination of the component IDs is a combination in which the arrangement order of the components 5 is considered, and the respective components 5 are joined only at the junction reference point 6, but even if they are junctions of the same components 5 Are different junctions. For example, when the part 5 has four parts A to D, and each part 5 has two joint reference points 6 at both ends, all combinations of part IDs are combinations arranged using all the parts A to D There are 384 (4! × 2 ⁴ ) combinations. In this case, a partial combination of the part IDs is a combination in which a part of the parts A to D is taken out and the arrangement order of the parts 5 is further taken into consideration. For example, a partial combination of part IDs includes 8 (2! × 2 ² ) combinations for taking out and arranging parts B and C of parts A to D, and a part A of parts A to D, Include 48 (3! × 2 ³ ) combinations for taking out and placing B and C.

（式１０）
で表すことができる。このようにクエリ再生成決定部３８は、クエリを再生成する回数を有限にすることができる。なお上記実施例においては、すべての部品５が２つの接合基準点を有し、構造物は部品データベース４１が記憶するすべての部品５から構成されるものとしたが、これは説明を簡単にするための１つの例示である。構造物はｎ個すべてを使用しなくてもよいし、部品５が有する接合基準点６は２つでなくてもよい。 In one example, the query regeneration determination unit 38 compares the quantity NQ of the query storage data stored in the history database 42 with the quantity NP of all combinations of component IDs stored in the component database 41. The query regeneration determination unit 38 determines to regenerate the query if the quantity NQ is less than the quantity NP, and determines not to regenerate the query otherwise. For example, it is assumed that the number of parts 5 stored in the parts database 41 is n, each part 5 has joint reference points 6 at both ends, and the structure comprises all parts 5 stored in the parts database 41. Think. In this case, the total number of queries that can be generated by the query generation unit 32, that is, the quantity NP of all combinations of part IDs stored in the parts database 41 is based on the quantity of the arrangement pattern of parts 5 and the quantity of junction reference point 6.

(Equation 10)
Can be represented by In this manner, the query regeneration determination unit 38 can make the number of times of regeneration of the query finite. In the above embodiment, all the parts 5 have two joint reference points, and the structure is made up of all the parts 5 stored in the parts database 41, but this simplifies the explanation. It is an example for one. The structure may not use all n pieces, and the joint reference point 6 which the part 5 has may not be two.

  In one example, the image recognition unit 31 calculates the certainty when recognizing the part ID. In this example, a predetermined one of combinations of part or all of the part IDs stored in the part database 41 includes the part IDs recognized by the image recognition unit 31 with a certainty or more certainty.

  In one example, the component database 41 stores priority information on the priority of the component 5 associated with the component ID in association with the component ID. In the present example, a predetermined one of combinations of part or all of the part IDs stored in the part database 41 includes the part IDs of priorities equal to or higher than a predetermined value recognized by the image recognition unit 31.

  When the query regeneration determination unit 38 determines to regenerate a query, the query regeneration unit 39 generates a query including data different from the query storage data stored in the history database 42. For example, when the number of parts 5 stored in the parts database 41 is n, each part 5 has two joint reference points 6 at both ends, and the structure is composed of all the parts 5 stored in the parts database 41 think of. The query regeneration unit 39 generates a query that does not include the combination of the component IDs included in the query storage data stored in the history database 42 among the queries that can be generated by the query generation unit 32 represented by Equation 10.

  In one example, when the length between the three-dimensional model of two parts 5 of the parts 5 arranged in the virtual space by the part arrangement unit 34 is equal to or more than a predetermined value, the query regeneration unit 39 determines the length The query is generated by adding the component ID of the component 5 corresponding to. In one example, for a part ID whose recognition accuracy by the image recognition unit 31 is less than a predetermined value or a part ID determined to have no consistency, a query in which the position or posture of the part ID is changed or other parts Prioritize queries that have been replaced with IDs. Thereby, it is possible to make the information processing of the three-dimensional recognition system 1 more efficient.

  Next, information processing of the stereoscopic recognition system 1 according to the embodiment of the present invention will be described using the flowchart shown in FIG. The information processing illustrated in FIG. 10 is realized by causing the electronic device 3 to execute a program and causing the server 4 to execute the program. In this example, the query storage data uses the query itself.

  In step 101, the server 4 deletes the query stored in the history database 42 before the electronic device 3 recognizes the image. As a result, the server 4 initializes the history database 42 and puts the history database 42 in a state where no query is stored.

  Next, in step 102, the image recognition unit 31 extracts a marker that can identify each component 5 from the image captured by the imaging device 14, and the extracted marker and component ID associated with each of the components 5. The position data of the part 5 is acquired. At this time, the image recognition unit 31 recognizes, from the image data captured by the imaging device 14, the base portion that forms the base portion on which the structure is placed.

  Next, in step 103, the query generation unit 32 generates a query including each of the component IDs obtained in step 102 and position data subordinate to each of the component IDs.

  Next, in step 104, the data acquisition unit 33 acquires, from the component database 41, the three-dimensional model of the component 5 and the constraint data associated with each of the component IDs included in the query, using the generated query.

  Next, in step 105, the component placement unit 34 places each component 5 in the virtual space associated with the real space using the position data and the three-dimensional model for each component ID included in the query. In addition, the component placement unit 34 places the base in the virtual space. Subsequently, the component placement unit 34 identifies a set of two adjacent components 5 among the disposed components 5 by determining whether or not each of the disposed components 5 is adjacent. The component placement unit 34 selects a pair of two components 5 in a predetermined order from the pair of two components 5 identified as being adjacent. The component placement unit 34 adjusts the position by moving one of the two components 5 or the like so as to satisfy the bonding restriction condition in each of the two sets of the components 5 identified as being adjacent to each other. And determine the placement.

  Next, in step 106, the consistency determination unit 35 determines the position of each placed part 5 based on the positional relationship of each part 5 placed in step 105, specifically by at least one of the following determinations. Determine the consistency of From the positional relationship between the position of each of the arranged parts 5 and the position of the base portion, the consistency determining unit 35 determines that there is a consistency when the arranged parts 5 are arranged according to (a) gravity. (B) If the arrangement is not in accordance with gravity, it is determined that there is no consistency. Furthermore, the consistency determination unit 35 determines that there is consistency when the overlap of the area occupied by the three-dimensional model of each of the arranged parts 5 is (a) within a predetermined volume, and (b) a predetermined volume If it exceeds, it is determined that there is no consistency. Furthermore, the consistency determination unit 35 determines that the degree of coincidence between the image captured by the imaging device 14 and the image viewed from the virtual camera of each component 5 placed in the virtual space by the component placement unit 34 is (a) a predetermined value or more If it is, it is determined that there is consistency, and if it is less than (b) a predetermined value, it is determined that there is not consistency.

  If it is determined in step 106 that all (a) consistency is present, the process proceeds to step 107, and the three-dimensional recognition unit 36 recognizes a structure based on the position of each part 5 arranged in step 105, This process ends.

  If it is determined in step 106 that at least one (b) is not consistent, the process proceeds to step 108. In step 108, the database storage unit 37 stores the query used in step 104 in the history database 42, and the process proceeds to step 109.

  In step 109, the query regeneration determination unit 38 determines whether the query stored in the history database 42 includes a predetermined combination of component IDs of each of the plurality of components 5 stored in the component database 41. If the history database 41 does not include a query including a predetermined combination of component IDs, the query regeneration determination unit 38 determines to regenerate the query, and the process proceeds to step 110. When the history database 41 includes a query including a predetermined combination of component IDs, the query regeneration determination unit 38 determines not to regenerate a query, and the process ends. The electronic device 3 can output an error to the display device 12 or the like when it is determined that the query is not regenerated and the process ends.

  In step 110, the query regeneration unit 39 generates a query different from the query stored in the history database 42, and the process proceeds to step 104. When the length between the three-dimensional model of two parts 5 of the parts 5 arranged in the virtual space by the part arrangement unit 34 is equal to or more than a predetermined value, the query regeneration unit 39 calculates the parts 5 corresponding to the length Generate queries with priority added to the component ID of.

  Next, the effect of the three-dimensional recognition system 1 according to the embodiment of the present invention will be described. In the present embodiment, the three-dimensional recognition system 1 targets a structure created by combining the parts 5 prepared in advance, and for each of the parts 5, a joint constraint that indicates how to join the three-dimensional model of the part 5 Are pre-defined, and these data are stored in the parts database 41. The three-dimensional recognition system 1 receives as an input a photographed image photographed using a standard smartphone camera, and individually recognizes elements (parts 5) constituting a structure without directly recognizing a three-dimensional shape from the photographed image. Independently detect. The three-dimensional recognition system 1 generates a query including the component ID of the component 5 detected individually, acquires data from the component database 41 using the query, and satisfies the joining constraint condition, so that the three-dimensional configuration of each component 5 Join models and place them on virtual space. The three-dimensional recognition system 1 evaluates the physical consistency of the part arrangement, and repeats the search and alignment on the part database 41 while updating the query until the appropriate arrangement is obtained.

  With such a configuration, in the present embodiment, the three-dimensional recognition system 1 can reconstruct a three-dimensional model of the entire structure from a single image and recognize its three-dimensional shape. In particular, when directly recognizing from a single image, it is possible to estimate spatial information of a part with few external surface exposures, which can not be handled by the existing three-dimensional shape recognition technology, and to recognize the three-dimensional shape of the entire structure. Thus, in the present embodiment, it is possible to achieve both the simplicity of the user's operation and the freedom in creating the structure when recognizing a three-dimensional shape from a planar image obtained by photographing the structure.

  Further, by adopting such a configuration, in the present embodiment, the three-dimensional recognition system 1 has a function of automatically correcting the position of the component 5 by defining the restriction on the bonding position of the component 5 using the bonding constraint condition. Have. As a result, noise generated in the assembly process of the component 5 and the image processing process can be alleviated, and the structure can be recognized more stably.

  Further, by adopting such a configuration, in the present embodiment, since the three-dimensional recognition system 1 recognizes only the known part 5, machine learning for recognition of the part 5 is not necessary, and the calculation cost can be reduced. It becomes possible.

  Further, by adopting such a configuration, in the present embodiment, since the three-dimensional recognition system 1 does not rely on a special sensor, only the image taken using a standard smartphone camera is used as an input. Since no special equipment is required, it can be realized inexpensively. Further, since any part of the component 5 may be used as a rigid body as long as it is difficult to change its shape, it can be manufactured inexpensively in accordance with the application. For example, select an appropriate material according to the application, such as using wood for applications where a strong impact such as throwing or stepping is expected, and using paper for applications that include up to the creation of the part 5 up to the user. It is possible.

  Further, in the present embodiment, when repeating the search and the alignment while updating the query, the three-dimensional recognition system 1 stores the history of the query in the history database 42, thereby preventing the search using the same query. With such a configuration, it is possible to limit the number of times the query is regenerated and stop the processing of the three-dimensional recognition system 1 appropriately. Further, since the history database 42 is used to prevent the search using the same query, the three-dimensional recognition system 1 deletes the query stored in the history database 42 before starting the recognition process of the structure. Do.

  Further, in the present embodiment, the three-dimensional recognition system 1 determines whether or not the length between the bonding reference points 6 of the two parts 5 initially disposed in the virtual space is within a predetermined value, and within a predetermined value. In some cases, it is determined that the two parts 5 are adjacent. With such a configuration, the three-dimensional recognition system 1 adjusts the position of the set of two components 5 within a predetermined value so as to satisfy the bonding restriction condition, and performs bonding for the other components 5. It is possible to prevent position adjustment so as to satisfy the constraint conditions. As a result, when a part 5 is not detected from the photographed image, etc., position adjustment is performed on the two parts 5 disposed so as to sandwich the part 5 so as to satisfy the bonding restriction condition. It is possible to prevent and reduce mismatch detection.

  Further, in the present embodiment, the marker is an AR marker, and is constituted of, for example, a rectangular black frame and an asymmetric dot pattern drawn inside thereof, and the dot pattern is configured to be different depending on each component 5. Thereby, using the known image recognition technology, the stereoscopic recognition system 1 obtains the position and orientation of the marker from the photographing device 14 from, for example, the shape and size of the rectangular black frame in the image in which the marker is photographed. It is possible to The three-dimensional recognition system 1 can identify each part 5 from, for example, a dot pattern in an image in which a marker is photographed, using a known image recognition technology.

  Further, in the present embodiment, the query includes a part ID and data subordinate to the part ID, and the data subordinate to the part ID corresponds to position data of the part 5 associated with the part ID, a three-dimensional model, And constraint data. When a query is generated, the three-dimensional recognition system 1 uses the query to generate a three-dimensional model and constraint data of each of the components 5 associated with each component ID included in the query, using the query. To each of the queries. Thus, the three-dimensional recognition system 1 acquires the three-dimensional model and the constraint data of each of the parts 5 from the parts database 41. As described above, in the present embodiment, the three-dimensional recognition system 1 performs a search in the component database 41, data acquisition from the component database 41, and storage of data in the history database 42 using a query. Thereby, it is possible to make the information processing of the three-dimensional recognition system 1 more efficient.

  Further, in the present embodiment, the three-dimensional recognition system 1 determines the consistency of the positions of the arranged parts 5 based on the overlapping degree of the area occupied by the three-dimensional model of the arranged parts 5. Further, in the present embodiment, the three-dimensional recognition system 1 determines whether or not the arranged components 5 are arranged in accordance with gravity from the positional relationship between the arranged components 5 and the position of the base portion. Do. Further, in the present embodiment, the three-dimensional recognition system 1 is based on the matching degree between the image photographed by the photographing device 14 and the image viewed from the virtual camera of each component 5 disposed in the virtual space by the component placement unit 34. , Determine consistency. With such a configuration, in the present embodiment, the three-dimensional recognition system 1 can evaluate the physical consistency of the component arrangement. Thereby, the three-dimensional recognition system 1 can determine whether or not the recognized structure is appropriate.

  Further, in the present embodiment, the three-dimensional recognition system 1 determines whether the query stored in the history database 42 includes a predetermined combination of component IDs stored in the component database 41. If the predetermined combination of the component IDs includes the component ID recognized by the image recognition unit 31 with a certainty or more certainty or the component ID with a certain priority or more, the type of query to be generated may be further limited. It becomes possible. Thereby, it is possible to make the information processing of the three-dimensional recognition system 1 more efficient. When the predetermined combination of the part IDs is all combinations, it is possible to generate all the queries that can be generated based on the parts database 41, and the number of times of regenerating the queries is limited. It is possible to stop the processing appropriately.

  In the present embodiment, when it is determined that the three-dimensional recognition system 1 regenerates the query, the length between the three-dimensional models of two parts 5 of the parts 5 arranged in the virtual space is a predetermined value or more , And generates a query to which the part ID of the part 5 corresponding to the length is added. With such a configuration, in the present embodiment, the component 5 that could not be recognized from the captured image is estimated, and the information processing of the three-dimensional recognition system 1 is preferentially generated by preferentially generating the query to which the component 5 is added. Can be made more efficient.

  The above-described effects are the same in the other embodiments and the other examples unless otherwise stated.

  In another embodiment of the present invention, a program for realizing the information processing shown in the functions and flowcharts of the embodiments of the present invention described above and a computer readable storage medium storing the program can also be used. In another embodiment, the electronic apparatus may independently implement the information processing shown in the functions and flowcharts of the embodiments of the present invention described above. Another embodiment can be a method for realizing the information processing shown in the function or the flowchart of the embodiment of the present invention described above. In another embodiment, a server that can supply a computer with a program that implements the information processing shown in the functions and flowcharts of the embodiments of the present invention described above can also be used. In another embodiment, a virtual machine can be realized that implements the information processing shown in the functions and flowcharts of the embodiments of the present invention described above.

  Hereinafter, modified examples of the embodiment of the present invention will be described. The modifications described below can be appropriately combined and applied to any embodiment of the present invention as long as no contradiction arises.

  In one example, the alignment metadata of one part 5 is determined by the distance and angle condition of the joint reference point of any other part with respect to the joint reference point of one part 5; It is data that defines the constraints of the part 5 and any other one part 5. In the present example, the part database 41 stores one alignment metadata, which defines a constraint between one part 5 and an arbitrary part 5 to be joined, in association with the part ID of the one part 5. In this example, the constraint data of one component 5 includes one alignment metadata, and the alignment metadata includes the position of the bonding reference point 6 and the data subordinate to the bonding reference point 6. Constraint and Posture Constraint elements.

  In one example, the component placement unit 34 determines whether the length between the center points of the two components 5 initially placed is within a predetermined value, and if within a predetermined value, the two components 5 is determined to be adjacent. In another example, the component placement unit 34 determines whether the length between the rigid bodies of the two components 5 initially placed is within a predetermined value, and if within a predetermined value, the two components 5 Is determined to be adjacent. In any case, the component placement unit 34 selects a set of components 5 in a predetermined order from a set of two components 5 determined to be adjacent, and in each set, satisfies the joining constraint condition. , Move one of the two parts 5 or the like to determine the arrangement.

  In one example, when it is determined in step 106 that the consistency determination unit 35 determines that there is no consistency, the consistency determination unit 35 calculates an evaluation value of the consistency of the position of each of the parts 5 arranged by the parts arrangement unit 34. In step 108, the database storage unit 37 further stores the evaluation value of the consistency of the position of each component 5 arranged by the component arrangement unit 34 in the history database 42 in association with each of the component IDs included in the query storage data. . If it is determined in step 109 that the query regeneration determination unit 38 does not regenerate the query, the electronic device 3 determines, in step 105, each component arranged in the virtual space using the query having the highest consistency with the evaluation value. The structure is recognized by the three-dimensional model of 5 and position data. Thereby, the electronic device 3 uses the arrangement of the parts 5 having the highest consistency according to the evaluation value among the arrangements of the parts 5 determined to be incoherent, and a three-dimensional model of the whole structure It is possible to recognize

  In the processing or operation described above, in a step, processing or operation is free as long as no processing or operation contradiction occurs such as using data which should not be available in the step. It can be changed. The embodiments described above are merely examples for explaining the present invention, and the present invention is not limited to these embodiments. The present invention can be practiced in various forms without departing from the scope of the invention. For example, the external shape of each device of each device is not limited to that illustrated.

Reference Signs List 1 stereo recognition system 2 network 3 electronic device 4 server 5 component 6 joint reference point 11, 21 processor 12, 22 display device 13, 23 input device 14 imaging device 15, 24 storage device 16, 25 communication device 31 image recognition unit 32 query Generation unit 33 Data acquisition unit 34 Component placement unit 35 Consistency judgment unit 36 Solid recognition unit 37 Database storage unit 38 Query regeneration determination unit 39 Query regeneration unit 41 Component database 42 History database

Claims (19)

A program for recognizing a solid in a real space composed of a plurality of predetermined parts, the program being executed in an electronic device provided with a photographing device for photographing the solid, To electronic devices,
An image recognition step of extracting features capable of identifying each part from the image captured by the imaging device, and acquiring each ID of the part and position data of the part based on the feature;
A query generation step of generating a query including an ID of each of the parts acquired based on the feature;
When a query is generated, the query includes the component database that stores the ID of each component, the three-dimensional model of each component, and the constraint data indicating the junction constraint between each component using the query. A data acquisition step of acquiring a three-dimensional model and constraint data of each of the parts associated with each of the IDs;
A component arrangement step of arranging each component in a virtual space corresponding to the real space based on the acquired position data of each component, the three-dimensional model, and the constraint data;
A consistency determination step of determining the alignment of the positions of the arranged parts based on the positional relationship of the arranged parts;
A three-dimensional recognition step of recognizing a three-dimensional object based on the position of each of the arranged parts when it is determined that the consistency is present;
If it is determined that the consistency is not present,
A database storage step of storing, in a history database, query storage data including at least each component ID among the queries used in the data acquisition step; query storage data stored in the history database and data stored in the components database A query regeneration decision step of deciding whether to regenerate the query based on
A query regenerating step of generating a query including data different from the query storage data stored in the history database, when it is determined to regenerate the query;
To run the program. The program is stored in the electronic device
The program according to claim 1, wherein before the image recognition step, a database initialization step of deleting the query storage data stored in the history database is executed.   The component arranging step arranges each component in the virtual space using the acquired position data of each component and the three-dimensional model, and the joint constraint condition is satisfied in each of the adjacent components in the virtual space. The program according to claim 1 or 2, wherein the position of each part is adjusted and arranged.   In the component placement step, whether or not two components are adjacent based on whether or not a length between three-dimensional models of two components of the components disposed in the virtual space is within a predetermined value. The program according to claim 3, which determines   The program according to any one of claims 1 to 4, wherein the position is a position with six degrees of freedom.   The program according to any one of claims 1 to 5, wherein the feature that can identify each part is an AR marker that each part has on the part surface.   When the query is generated, the data acquisition step uses the query to correspond the three-dimensional model and constraint data of each of the components associated with each of the IDs included in the query to the ID using the query. The program according to any one of claims 1 to 6, which is obtained by adding each to the query.   The consistency judgment step determines the consistency of the position of each of the arranged parts based on the overlapping degree of the area occupied by the three-dimensional model of each of the arranged parts. The program described in Section 1. The image recognition step recognizes a base portion that forms a base portion of the solid from the image captured by the imaging device,
The component placement step places the base portion on the virtual space,
The consistency determination step determines whether or not each of the arranged parts is arranged according to gravity from the positional relationship between the positions of the arranged parts and the position of the base portion, and the judgment is made on the basis of the judgment. The program according to any one of claims 1 to 8, wherein the consistency is further determined.   In the consistency determination step, an image captured by the imaging device and an image of each of the disposed components viewed from a virtual camera disposed at the position of the virtual space corresponding to the position of the imaging device The program according to any one of claims 1 to 9, wherein the consistency is determined based on the degree of coincidence.   The database storage step according to any one of claims 1 to 10, wherein the position of each of the arranged parts is further stored in the history database in association with the ID of each part included in the data for query storage. Programs.   Whether the query regeneration determination step regenerates a query based on whether or not the query storage data stored in the history database includes a predetermined combination of the IDs of the components stored in the components database. The program according to any one of claims 1 to 11, which determines. The program is stored in the electronic device
The program according to any one of claims 1 to 12, further causing an error output step of outputting an error if it is determined that the query is not regenerated.   The database storage step further stores an evaluation value of the consistency of the position of each of the arranged parts in the history database in association with the ID of each of the parts included in the data for storing a query. The program according to any one of the items. The program is stored in the electronic device
The second three-dimensional recognition step of recognizing a three-dimensional object based on the position of each of the arranged parts having the highest consistency by the evaluation value is executed when it is determined that the query is not regenerated. Described program.   In the query regenerating step, when a length between a three-dimensional model of two parts of two parts arranged in the virtual space is equal to or more than a predetermined value, the ID of the part corresponding to the length is added The program according to any one of claims 1 to 15, which generates a query. A three-dimensional recognition system for recognizing a three-dimensional object in a real space composed of a plurality of predetermined parts, comprising:
An electronic device comprising a photographing device for photographing the three-dimensional object;
A part database that associates and stores ID of each part, a three-dimensional model of each part, and constraint data indicating joining constraints among the parts;
A history database storing query storage data including at least the ID of each of the queries input to the components database;
An image recognition unit which extracts features capable of identifying each part from the image captured by the imaging device, and acquires an ID of each part and position data of the part based on the feature; and A query generation unit that generates a query including the ID of each of the acquired parts;
When a query is generated, using the query, a data acquisition unit that acquires, from the component database, a three-dimensional model of each of the components associated with each of the IDs included in the query and constraint data;
A component placement unit for placing each component in a virtual space corresponding to the real space based on the acquired position data of each component, the three-dimensional model, and the constraint data;
A consistency determination unit that determines the alignment of the positions of the arranged parts based on the positional relationship of the arranged parts;
A three-dimensional recognition unit that recognizes a three-dimensional object based on the position of each of the arranged parts when it is determined that the consistency is present;
If it is determined that the consistency is not present,
A database storage unit for storing, in the history database, query storage data including at least an ID of each of the components used by the data acquisition unit;
A query regeneration determination unit that determines whether to regenerate the query based on the query storage data stored in the history database and the data stored in the component database, and it is determined to regenerate the query A query regenerating unit that generates a query including data different from the query storage data stored in the history database;
A three-dimensional recognition system comprising An electronic device for recognizing a solid in a real space composed of a plurality of predetermined parts, the electronic device comprising:
A photographing device for photographing the solid;
A part database that associates and stores ID of each part, a three-dimensional model of each part, and constraint data indicating joining constraints among the parts;
A history database storing query storage data including at least the ID of each of the queries input to the components database;
An image recognition unit which extracts features capable of identifying each part from the image captured by the imaging device, and acquires an ID of each part and position data of the part based on the feature; and A query generation unit that generates a query including the ID of each of the acquired parts;
When a query is generated, using the query, a data acquisition unit that acquires, from the component database, a three-dimensional model of each of the components associated with each of the IDs included in the query and constraint data;
A component placement unit for placing each component in a virtual space corresponding to the real space based on the acquired position data of each component, the three-dimensional model, and the constraint data;
A consistency determination unit that determines the alignment of the positions of the arranged parts based on the positional relationship of the arranged parts;
A three-dimensional recognition unit that recognizes a three-dimensional object based on the position of each of the arranged parts when it is determined that the consistency is present;
If it is determined that the consistency is not present,
A database storage unit for storing, in the history database, query storage data including at least an ID of each of the components used by the data acquisition unit;
A query regeneration determination unit that determines whether to regenerate the query based on the query storage data stored in the history database and the data stored in the component database, and it is determined to regenerate the query A query regenerating unit that generates a query including data different from the query storage data stored in the history database;
An electronic device comprising: A method implemented in an electronic device comprising a photographing device for photographing a solid, for recognizing a solid in a real space composed of a plurality of predetermined parts.
An image recognition step of extracting features capable of identifying each part from the image captured by the imaging device, and acquiring each ID of the part and position data of the part based on the feature;
A query generation step of generating a query including an ID of each of the parts acquired based on the feature;
When a query is generated, the query includes the component database that stores the ID of each component, the three-dimensional model of each component, and the constraint data indicating the junction constraint between each component using the query. A data acquisition step of acquiring a three-dimensional model and constraint data of each of the parts associated with each of the IDs;
A component arrangement step of arranging each component in a virtual space corresponding to the real space based on the acquired position data of each component, the three-dimensional model, and the constraint data;
A consistency determination step of determining the alignment of the positions of the arranged parts based on the positional relationship of the arranged parts;
A three-dimensional recognition step of recognizing a three-dimensional object based on the position of each of the arranged parts when it is determined that the consistency is present;
If it is determined that the consistency is not present,
A database storage step of storing, in a history database, query storage data including at least each component ID among the queries used in the data acquisition step; query storage data stored in the history database and data stored in the components database A query regeneration decision step of deciding whether to regenerate the query based on
A query regenerating step of generating a query including data different from the query storage data stored in the history database, when it is determined to regenerate the query;
Method, including.

Images