JP4624000B2 - Compound artificial intelligence device - Google Patents

Description

The present invention relates to an artificial intelligence apparatus that automatically controls an arbitrary object such as a camera, a vehicle, or a robot based on a moving image captured by a video camera or the like.
In particular, the present invention can improve the recognition rate by recognizing an arbitrary target portion after recognizing a three-dimensional space in which the artificial intelligence device itself exists, and a three-dimensional object is prepared in advance. By replacing with corresponding 3D parts, the recognition rate of the object can be dramatically improved, and by sharing the recognition information among multiple artificial intelligence devices, information from multiple directions can be integrated. The present invention relates to a composite artificial intelligence device that can dramatically improve recognition accuracy by making a judgment.

In general, an artificial intelligence device is a technology related to the realization of artificial vision of artificial intelligence. Artificial intelligence often relies on visual information, and is largely a technology belonging to the field of image recognition.
In artificial intelligence, it is necessary to first grasp the entire space and then recognize the object. Artificial intelligence technology also requires technology for projecting the real world to the virtual world by developing image recognition technology.
By using such artificial intelligence as a means for recognition and determination, for example, it is possible to realize automatic control and automatic driving for an arbitrary target such as a monitoring camera, a vehicle, or a work robot.

Until now, various artificial intelligence techniques for recognizing a three-dimensional object by image recognition have existed, but no definitive techniques have yet existed.
As a conventional artificial intelligence technique using image recognition, for example, a “fire accusing guidance device” in which artificial intelligence determines and displays an evacuation guidance route based on a fire image captured by a surveillance camera (Patent Document 1) In addition, there is an “object recognition and tracking system” (Patent Document 2) equipped with artificial intelligence that detects and recognizes an object from an image obtained by a video camera or the like.

Japanese Patent Laid-Open No. 08-122404 JP 2002-536646 A

With the conventional artificial intelligence technology using such image recognition, it is difficult to directly recognize a three-dimensional object, and it has not been possible to cope with an actual condition in which the shape changes from moment to moment.
In addition, there is a limit in recognizing an object that is originally three-dimensional only in two dimensions (images), and it is extremely difficult to recognize an actual three-dimensional object efficiently and accurately with conventional artificial intelligence technology. It was difficult.
In addition, it is difficult to perform consistent recognition and determination for the entire system and control of an object based on the determination based on individual determination by a single artificial intelligence device.

Here, as a technology for recognizing a target three-dimensionally based on two-dimensional image information, three-dimensional coordinates of a plurality of points in the two-dimensional image are obtained, and a rough shape of the target is generated from the distribution of the three-dimensional points. Technology has been proposed.
However, it is impossible to generate a three-dimensional shape from only a set of points. Actually, the accuracy is poor and the three-dimensional shape cannot be accurately reproduced.
Further, only the two-dimensional image information from the observer's camera cannot obtain information on the hidden part of the object, and eventually only the three-dimensional shape of a part of the object is obtained.

  The present invention has been proposed in order to solve the above-described problems of the conventional technology, and the recognition rate is obtained by recognizing an arbitrary target portion after recognizing the three-dimensional space in which the artificial intelligence device itself exists. In addition, it is possible to dramatically improve the recognition rate of objects by replacing them with corresponding three-dimensional parts prepared in advance. It is an object of the present invention to provide a composite artificial intelligence device that can significantly improve recognition accuracy by comprehensively judging information from multiple directions by sharing recognition information between intelligent devices.

In order to achieve the above object, the present invention includes a plurality of artificial intelligence devices attached to an arbitrary object (object), and a composite artificial intelligence device is formed by a plurality of artificial intelligence devices arranged at arbitrary positions. Constitute. Thereby, recognition accuracy is raised by comprehensively judging information from multiple directions.
Each artificial intelligence device obtains the attribute of the target object from the artificial intelligence device to which the object belongs, by giving information about its own attribute to other artificial intelligence devices, or vice versa. Thus, the determination accuracy is dramatically increased.
In the present invention, three-dimensional space recognition and three-dimensional object recognition are separated, first the three-dimensional space is recognized, the relationship between itself and the surroundings is clarified, and then the coordinates and viewing angles of the object are narrowed down and recognized. In this way, the recognition rate is increased.

In addition, in the case of a 3D shape object that cannot be recognized by the 2D shape of the image, the 3D shape is returned to the 3D shape as necessary and compared with the 3D data (parts) prepared in advance. Try to replace it. Thus, by replacing the object with a three-dimensional part prepared in advance, the object can be recognized and its three-dimensional shape can be recognized at the same time, and the recognition rate can be dramatically improved.
Replacement with a 3D part generates a polygon from a set of points in a 2D image, generates a rough shape of a 3D object, and compares the 3D surface with one side of the 3D part. This is done by specifying the part. Further, in the process of generating the three-dimensional surface, the shape generation is performed by adding a part for which details are desired.

Specifically, feature points in the image are extracted and tracked, their three-dimensional coordinates are obtained, and polygons based on the feature points are generated. The surface shape is extracted as a polygon, the surface of the polygon is compared with the surface of the component three-dimensionally, and the component is specified by comparing with the prepared three-dimensional component.
Since attributes are given to the parts, the current meaning can be understood from the relationship of the attributes, and the overall situation and individual situation can be understood. And the understanding of those meanings is performed among a plurality of artificial intelligence devices, and comprehensive judgment is made.
As a result, it is possible to make a single judgment based on each artificial intelligence measure that is consistent with the overall judgment, and to determine the next action.

As described above, in the composite artificial intelligence apparatus of the present invention, the real world is projected onto the virtual space in real time, the surrounding space where the user exists is first recognized, and then the object is recognized.
The information obtained by the plurality of artificial intelligence devices is shared with each other to realize comprehensive space recognition and object recognition. Even with a single artificial intelligence device, it is possible to recognize an object, specify a position, specify a shape, understand / judge the meaning, and control the object, but it is a complex artificial intelligence device with multiple artificial intelligence devices Network environment, and by exchanging information and judgment results of individual artificial intelligence devices with each other, comprehensive judgment becomes possible, and it is more accurate and wider than that of a single artificial intelligence device. Spatial recognition and object recognition are possible.
As a result, the real world can be accurately and effectively replaced with a virtual space, and by using this in each artificial intelligence device, it is possible to make decisions and actions in each artificial intelligence device consistent with the overall judgment.

Further, in the present invention, each artificial intelligence apparatus is placed in its own position by making rules on the relationship between the arrangement of the three-dimensional data (parts) to be replaced with the recognized countermeasures and the attributes in advance. Can understand and decide the next action.
In each of the individual artificial intelligence devices, surrounding space recognition can be realized by obtaining a three-dimensional distribution of feature points in an image captured from the own camera and simultaneously obtaining the own camera position. A plurality of artificial intelligence devices are integrated by integrating the coordinates.

The recognition of the target object is possible with a single artificial intelligence device, but once the space is recognized and the approximate position and orientation are grasped, it is performed while exchanging information with each other with the composite artificial intelligence device. Object recognition, which has been difficult until now, becomes easy and accurate.
All recognized objects are replaced with parts with known attributes, and a virtual space is created by the parts, so each artificial intelligence device grasps its position and situation in the virtual space and makes sense With this understanding, the next action can be determined.

As described above, according to the present invention, by providing an artificial intelligence device in a main object, a detailed three-dimensional position and orientation is acquired for the positional relationship and shape between itself and another object, and the object ( Artificial intelligence devices) can act independently by judging the situation. The main object is, for example, a vehicle, furniture in a house, a wheelchair, or a human, but a plurality of artificial intelligence devices can be attached to each object to exchange information.
Each artificial intelligence device can grasp its own position, convey its position and orientation to surrounding objects, and have a common understanding in a unified coordinate system.
Each artificial intelligence device alone can recognize and judge only a part of the space, but in a composite artificial intelligence device with a plurality of artificial intelligence devices attached to various objects, recognize the overall arrangement of the space where the object exists, 3D understanding and integration of 3D information obtained by a single artificial intelligence device, not only to obtain 3D images, but also by comparing the attribute information etc. of objects and objects in the image, I can grasp it.

Real objects are classified into objects having artificial intelligence, objects having a part of the functions, and objects not having a part of the functions. Even if there is an object that does not have a camera that forms artificial vision, a three-dimensional whole image can be obtained from other objects around the camera and an understanding of the real world can be obtained.
Then, the actual three-dimensional world that is understood is shared by each artificial intelligence device, that is, each object. As a result, the behavior conditions and the like are uniquely determined for each object, and the behavior can be integrated among the objects. As a result, each object can behave with a will, and the individual will and the overall will can be integrated with no contradiction.

For example, by attaching each artificial intelligence device according to the present invention to a car, it can be applied to automatic driving of a car, each vehicle can read traffic lights and road signs, and try to translate or overtake. The intention of the vehicle is transmitted to the surrounding related vehicles, and the artificial intelligence devices loaded on each vehicle determine the behavior without contradiction, thereby enabling automatic driving while ensuring safety among the vehicles.
In addition, information on the destination direction can be acquired from passing vehicles, and each vehicle can be reflected as a judgment material useful for automatic driving.

Thus, in the composite artificial intelligence apparatus of the present invention, it is possible to perform collaborative work using a plurality of artificial intelligence. Through the network of artificial intelligence devices, the recognition results in each artificial intelligence device can be integrated and information can be shared among all artificial intelligence devices.
Specifically, the identification result of the three-dimensional object can be shared by all the artificial intelligence devices. By sharing the acquired video information, an image at an invisible position can be acquired from other artificial vision and comprehensively determined. Parts can be identified only by comparison with attributes. The experience of each artificial intelligence device can be shared as the entire knowledge, and the entire knowledge can be generated using the entire knowledge, and individual judgments without contradiction to the entire intention can be made.
In the ubiquitous era, a PC is attached to everything, and the near future is depicted where information can be exchanged. The ultimate figure is the present invention.

Hereinafter, preferred embodiments of the composite artificial intelligence device according to the present invention will be described with reference to the drawings.
Here, the composite artificial intelligence apparatus of the present invention described below is realized by processing, means, and functions executed by a computer in accordance with instructions of a program (software). The program sends commands to each component of the computer, and performs the following predetermined processing and functions, such as automatic extraction of feature points from images, automatic tracking of extracted feature points, three-dimensional coordinates of feature points Calculation, camera vector calculation, three-dimensional part specification, coordinate specification, and the like. Thus, each process and means in the composite artificial intelligence apparatus of the present invention are realized by specific means in which the program and the computer cooperate.
Note that all or part of the program is provided by, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and the program read from the recording medium is installed in the computer and executed. The The program can also be loaded and executed directly on a computer through a communication line without using a recording medium.

[Basic functions]
First, basic functions realized by the composite artificial intelligence apparatus according to the present invention will be described with reference to FIGS.
FIG. 1 is a block diagram showing a basic functional configuration of each artificial intelligence device constituting the composite artificial intelligence device of the present invention.
FIG. 2 is an explanatory diagram illustrating a relationship between a real-world object and a target to which the artificial intelligence apparatus illustrated in FIG. 1 is attached.
FIG. 3 is an explanatory diagram showing more specifically the relationship between the object to which the artificial intelligence apparatus shown in FIG. 2 is attached and the target.

The artificial intelligence apparatus 100 shown in these drawings is an artificial intelligence apparatus that is attached to an arbitrary object such as a vehicle, furniture, or human, for example, and the plurality of artificial intelligence apparatuses 100 constitute the composite artificial intelligence apparatus according to the present invention. It has come to be.
As shown in FIG. 1, each artificial intelligence device 100 individually includes a self-object attribute storage function 101, a self-object spatial position and orientation recognition function 102, a self-object surrounding object recognition function 103, An object situation determination function 104, a processing process recording function 105, a communication function 106, and a continuation processing function 107 are realized.

Each artificial intelligence device 100 belonging to an arbitrary object has a communication function that enables communication with the artificial intelligence device 100 belonging to another object, and the attribute information of the self object obtained by each artificial intelligence device 100, self Image information around the object, analysis information of the image, spatial position and orientation recognition information of the self object, recognition information of the object around the self object, situation determination information around the self object, and some of these pieces of information or The combination information can be transmitted and transmitted to the artificial intelligence apparatus 100 belonging to another object, and the same information from the other artificial intelligence apparatus 100 can be received.
With such a communication function, the plurality of artificial intelligence devices 100 constitute a network, and each artificial intelligence device 100 receives the same information from the other artificial intelligence devices 100 as a recognition object, thereby performing recognition processing. The function of continuing processing by adding information from the artificial intelligence device of other objects taken in by the reception from the middle of the above can be realized.

In this way, each artificial intelligence device 100 can acquire image information, image analysis information, spatial position and orientation information, and surrounding information of each object acquired by the plurality of artificial intelligence devices 100 of the self object and other objects. Each coordinate system of object information is integrated into a common coordinate system, and it is more accurate than a single artificial intelligence device for a wider space and a wide range of objects, or with a single artificial intelligence device By dealing with unrecognizable objects, the real world can be transformed into a virtual world.
In the virtual world, spatial position and orientation recognition, object recognition, attribute recognition, etc. are performed, and comprehensive judgments that are difficult with a single artificial intelligence device are made. It is possible to perform situation judgment and action decision.

As an object to which the artificial intelligence device 100 is attached, as shown in FIG. 2A, among the objects existing in the real world, artificial sensors such as various sensors and video cameras, communication devices, computers, etc. are provided. It can be classified into two types: “an object to which an intelligent device is attached” (also simply referred to as “object”) and other “other objects” (also simply referred to as “objects”). In addition, as shown in FIG. 2 (b), “an object with an artificial intelligence device attached to the real world”, “an object with a part of the artificial intelligence device attached” (simply “other objects”), It can also be classified into three types of “other objects”. A plurality of artificial intelligence devices 100 may be attached to one object.
In this way, the plurality of artificial intelligence devices 100 attached to the plurality of objects recognize the objects around them, determine the situation, exchange information with other artificial intelligence devices 100, and determine the result. It is possible to form a composite artificial intelligence apparatus that exchanges not only information exchange but also information in the process of judgment processing to make a composite judgment.

The object to which the artificial intelligence device 100 is attached is individually the attribute storage function 101 of the self object, the spatial position and orientation recognition function 102 of the self object, the recognition function 103 of the object around the self object, and the process of these recognition functions. It has the function 105 which records. It also has a function 104 for determining the status of the self object based on the recognition.
The artificial intelligence apparatus 100 belonging to each object has a communication function 106 that enables communication with the artificial intelligence apparatus belonging to other objects, and this communication function 106 is obtained from the properties and sensors of the self object. Attribute information such as information, surrounding image information from the camera attached to the self object, information obtained by analyzing the image, information about the analysis process, 3D distribution map of the self object, and spatial position and orientation recognition information of the object to which the user belongs , The recognition information of the objects around the self object, and the situation determination information around the self object, the partial information or the combination information thereof is attached to the other object to the artificial intelligence apparatus 100 belonging to the other object. Transmit and receive from other artificial intelligence devices 100.

Each artificial intelligence device 100 can perform continuous processing by adding information from the artificial intelligence device 100 of other objects received by reception from the middle of the recognition processing (continuous processing function 107). Each coordinate system of each object acquired by each coordinate system to which the object belongs, the analysis information of the image, the spatial position and orientation information, and the surrounding object information are integrated into a common coordinate system. Thus, it is possible to deal with a wider range of space and a wider range of objects with higher accuracy than with the single artificial intelligence apparatus 100, or even an object that cannot be recognized by the single artificial intelligence apparatus 100. Thus, the object can be specified.
As a result, the real world can be converted into a virtual world, and situation determination and action determination can be performed by spatial position and orientation recognition, object recognition, attribute recognition, and the like, or by comprehensive judgment based on these recognitions.

Specific examples of the “object to which the artificial intelligence device is attached” include, for example, a vehicle to which the artificial intelligence device 100 is attached, a road incidental facility, and the like.
There are a plurality of vehicles in the real world, and among them, there are vehicles with an artificial intelligence device attached and vehicles without an artificial intelligence device 100, which are mixed and pass through a road.
In this way, the real world can be classified into two types: vehicles and road-related facilities with the artificial intelligence device 100 attached thereto, and other objects to which the artificial intelligence device 100 is not attached (see FIG. 2A).

In the communication between the artificial intelligence devices, not only the judgment result of the artificial intelligence device 100 but also the surrounding image obtained from the camera of the artificial intelligence device 100 attached to the vehicle as it is or in the state of the analysis process or analysis The result can be transmitted to the artificial intelligence apparatus 100 of another vehicle so that it is possible to determine a wide range of situations of the own vehicle even in a situation where it is far or invisible from the own vehicle.
In addition, the situation of the vehicle without the artificial intelligence apparatus 100 is captured by a camera provided in the other artificial intelligence apparatus 100, and the combined artificial intelligence apparatus 100 cooperates to display a single artificial It can be captured more accurately than by intelligent devices.
Furthermore, the situation of a distant vehicle that cannot be captured as an image can be obtained by communication between the composite artificial intelligence devices.

Here, the object provided with the artificial intelligence apparatus 100 may be not only a vehicle but also a road incidental facility, for example, a traffic light or a guard rail. Can also be replaced. Roads, roadside traffic lights, roadside trees, buildings, guardrails, road signs, and other road ancillary equipment can also be “objects with artificial intelligence devices” and “artificial intelligence devices are not attached” It can also be an “object”.
Each artificial intelligence apparatus 100 attached to a plurality of vehicles or road facilities can share information by determining the surrounding situation and expressing the determination result in common coordinates. And the vehicle to which the artificial intelligence apparatus 100 is attached can be automatically driven by combining with a control apparatus to be described later.
As an example other than a vehicle, for example, an artificial intelligence device 100 can be attached to an indoor main object such as a building or a moving object (wheelchair, robot, etc.) to exchange data in a LAN environment. . Wheelchairs and robots are automatically driven in a LAN environment and can take desired actions.

Further, as described above, as the object to which the artificial intelligence device 100 is attached, as shown in FIG. 2 (b), “ an object to which the artificial intelligence device is attached” and “a part of the functions of the artificial intelligence device are attached. It can also be classified into three types: “objects” and other “other objects”.
That is, the composite artificial intelligence device attaches some devices having a part of the function of the artificial intelligence device 100 to various objects to which the artificial intelligence device 100 is attached and other various other objects. Some devices belonging to the object alone do not have the function of the artificial intelligence device 100, but by transmitting information from some devices attached to other objects to the artificial intelligence device 100, or by the artificial intelligence device 100 By receiving the determination result, the function of the artificial intelligence apparatus 100 attached to various objects can be improved, or other objects can be controlled.

Specifically, as shown in FIG. 2B, an object in the “real world” is an “object with an artificial intelligence device attached” and an “object with a partial function of the artificial intelligence device”. And other “other objects”.
As a partial device provided with “a partial function of the artificial intelligence device”, for example, there is a device that has only a camera and an image transmission device and does not have an image recognition function. The functions provided in some devices and the functions omitted can be appropriately set according to the purpose of use or application target of the composite artificial intelligence device. However, some devices are always equipped with a communication device for transmitting the acquired information to another artificial intelligence device 100.
As described above, it is possible to acquire a wide range of information and to recognize an object with high accuracy even when using a part of the device that does not have all the functions of the artificial intelligence device 100 but has some functions. Moreover, the viewpoint with respect to one target object can be taken in multiple directions, and the recognition accuracy can be improved. Thereby, it is possible to simplify the configuration of a part of the apparatus while realizing accurate and highly accurate object recognition of the composite artificial intelligence apparatus, thereby realizing cost reduction and simplification of the entire apparatus.

As a specific example of “an object to which a part of the functions of an artificial intelligence device is attached”, for example, the artificial intelligence device 100 is attached to a part of a vehicle and road equipment, and some of the devices are used as traffic lights, roads, and street lights. Can be attached. Of course, some devices can be attached to the vehicle.
In this way, the real world is classified into a vehicle and an object to which the artificial intelligence device 100 is attached, a vehicle and an object to which some devices are attached, and a vehicle and an object to which nothing is attached.
Information sent from a part of the device is analyzed and analyzed by the artificial intelligence device 100, and is performed with higher efficiency and higher accuracy than the analysis and analysis judgment of the artificial intelligence device alone.
Since the vehicle to which the artificial intelligence device 100 is attached can determine the surrounding situation, the vehicle can be automatically operated by attaching the control device to the vehicle.

As shown in FIG. 3, the object “a” includes a video camera, various sensors such as an acceleration watch and a thermometer, an artificial intelligence device 100 including a communication device, and a wheel and a handle that move the position and direction of the object. It is a full-equipped object equipped with a control device that automatically controls, and can determine the situation by the artificial intelligence device 100 and drive the control device by a control signal.
The object “b” is an object that does not move by itself, such as an incidental facility on a road, and does not require a control device.
The object “c” includes only a part of the device and does not have a determination function, but has a communication function, and sends the data acquired by itself to the other artificial intelligence device 100 via the communication device. Information can be obtained from others. Therefore, when a judgment or the like is necessary, the other artificial intelligence device 100 is relied upon.
The object “d” is an object that is not provided with the artificial intelligence apparatus 100 or a part of the apparatus. In reality, an object (target object) such as the object “d” is mixed.

[Specific functions]
Next, more specific functions in the composite artificial intelligence apparatus of the present invention as described above will be described with reference to FIG.
FIG. 4 is a functional block diagram for explaining more specific functions realized in each artificial intelligence apparatus 100 shown in FIG.
As shown in the figure, the artificial intelligence device 100 realizes the following specific functions.

The three-dimensional space configuration unit 401 integrates image information and the like of the surrounding space acquired by each artificial intelligence device 100 and each partial device (see FIG. 2 and FIG. 3) into a unified coordinate system, and performs spatial tertiary Construct an outline of the original shape.
The object position / posture movement arrangement unit 402 includes the positions of various objects to which the artificial intelligence device 100 existing in the three-dimensional space of the space obtained by the three-dimensional space configuration unit 401 and other objects to which some devices belong. The posture and movement are acquired as 3D data and placed in a unified coordinate system.
The object position movement additional arrangement unit 403 acquires, as three-dimensional data, the three-dimensional position, posture, and movement of an object that does not belong to the object in the unified coordinate system in which the object is arranged by the object position / posture movement arrangement unit 402. , Place it in the unified coordinate system.

The database unit 404 stores a plurality of parts to which a three-dimensional shape and attributes corresponding to objects, other objects, and objects are given in advance.
The attitude narrowing unit 405a extracts parts corresponding to objects, other objects, and objects from candidate parts in the database unit 404, and predicts an approximate arrangement and attitude from the viewpoint position of the three-dimensional space configuration unit 401. , Narrow down the posture of the parts.
The component narrowing-down unit 405b by the attribute is held by the object after specifying or fixing the components (see the object target component specifying unit 407 and the object target component coordinate fixing unit 408) as described later, or at the same time or in advance. The attribute and the attribute of the part are compared, and if the attribute matches within the condition, the part candidate is narrowed down assuming that the specific possibility has increased.
The comparison unit 406 sequentially compares the narrowed parts with the three-dimensional shape data of a part of the object, other objects, and the target object.

The object target part specifying unit 407 determines whether the comparison in the comparison unit 406 matches or does not match, and specifies the corresponding part by matching.
The object target part coordinate fixing unit 408 determines the three-dimensional position of the part in detail at the same time as or after the specification by the object target part specifying unit 407 and rearranges it in the unified coordinate system.
The object part attribute comparison addition / update unit 409 adds or updates a new attribute to the part if the identification has already been completed.
The part reconstructed three-dimensional space generation unit 410 reconstructs a plurality of parts in a unified coordinate system to generate a part reconstructed three-dimensional space.

Real corresponding virtual space 411, the generated parts reconstructed three-dimensional space, giving an attribute to each part is a component, the real world, a virtual world by a plurality of components having attributes, the real corresponding virtual space Is generated.
The self-recognition unit 412 in the virtual space arranges its own three-dimensional position, posture, and movement in the generated reality-enabled virtual space, and obtains the three-dimensional arrangement and attribute relationship of the parts centered on the self-position. .
The semantic rule assigning unit 413 gives in advance a rule for generating a meaning generated from the three-dimensional arrangement of a plurality of parts and the mutual relationship of their attributes.
The meaning generation unit 414 generates a meaning from the positional relationship of itself and the relationship between these attributes in the reality corresponding virtual space.

The self-situation grasping unit 415 understands by replacing the situation placed in the three-dimensional space with the reality corresponding virtual space.
The time history simulation unit 416 confirms the self-situation obtained by the self-situation grasping unit 415 and surrounding relations by simulation from the past to the near future.
The behavior determination unit 417 determines the current behavior of the near future by checking and understanding the situation in the near future by the time history simulation unit 416.

The learning device 418 includes a process storage unit 418a and a type classification unit 418b, and reproduces a similar process experienced previously to simplify the process.
Specifically, the process storage unit 418a stores all or some processes leading to the action determination of the action determination unit 417 in a part or all of the artificial intelligence devices included in the composite artificial intelligence device.
The type classification unit 418b sorts and stores the processes in the process storage unit 418a into similar processes and stores them.
This simplifies the process by reproducing a similar process that was previously experienced.

The artificial intelligence apparatus 100 having specific functions as described above operates as follows.
As shown in FIG. 3, first, the image information of the surrounding space acquired by the individual artificial intelligence apparatus 100 by the three-dimensional space configuration unit 401, and also acquired by the individual artificial intelligence apparatus and each partial apparatus. The image information and the like of the surrounding space are integrated into the unified coordinate system, and an outline of the three-dimensional shape of the space is constructed.
Next, in the object position / posture movement / arrangement unit 402, the position, posture, and movement of various objects to which the artificial intelligence device 100 exists in the three-dimensional space of the space and other objects to which some devices belong are three-dimensional data. Is obtained by communication and placed in a unified coordinate system.

Next, the object position movement additional arrangement unit 403 acquires the three-dimensional position, orientation, and movement of the object that does not belong to the object as a three-dimensional data with respect to the unified coordinate system in which the object is arranged. Place additional.
In this way, all objects and objects are configured in a three-dimensional spatial arrangement including their movement.
Next, a three-dimensional part corresponding to the object and the object is extracted. A plurality of parts to which a three-dimensional shape and attributes corresponding to the object and the object are given in advance are stored in the database unit 404 in advance. In principle, this part has a three-dimensional CG indicating the shape, but it may be a two-dimensional part.

Next, the posture narrowing unit 405a narrows down the objects and other parts corresponding to the object and the target object from the candidate parts in the database unit 404, and predicts the approximate arrangement and posture from the viewpoint position of the three-dimensional space configuration unit 401. Then narrow down the posture of the parts. For example, if it is clear from the attribute of the object that it is in contact with the ground, it can be narrowed down to the posture arranged in parallel with the passage on the ground surface.
Next, the comparison unit 406 compares the three-dimensional shape data of the object, another object, and a part of the object with the narrowed parts one after another.
The comparison unit 406 performs attribute comparison such as three-dimensional comparison, two-dimensional comparison, color comparison, and comparison of data obtained from other sensors, and the object target component specifying unit 407 matches the comparison. And the corresponding parts are identified by matching. Here, it is possible to specify a part by matching and output and display it as a shape.

In the object target component coordinate fixing unit 408, the three-dimensional position of the component is determined in detail and rearranged in the unified coordinate system simultaneously with or after the specification of the component in the object target component specifying unit 407. Here, the coordinates of the identified part can be determined. Further, the shape and coordinates of the part can be displayed in the same coordinate system as the coordinate system by the three-dimensional space configuration unit 401. Further, by tracking a moving object or the like, it is possible to deal with a moving body.
In the component narrowing down part 405b by attribute, the attribute of the object is compared with the attribute of the component after specifying and fixing the component as described above, or at the same time or before, and the attribute is matched within the condition. For example, instead of specifying immediately, it is determined that the possibility of specifying has increased, and part candidates are narrowed down, and finally a pre-processing for specifying by shape comparison is performed.

If the part has already been specified, the object part attribute comparison addition / update unit 409 adds or updates new attributes such as temperature, stiffness, and color acquired by the sensor or the like to the specified part.
The component reconstructed three-dimensional space generation unit 410 generates a component reconstructed three-dimensional space by reconstructing the above-described unified coordinate system with a plurality of components. As a result, the real world is obtained as a virtual three-dimensional space reconstructed with three-dimensional parts. However, it is not necessary to replace all objects with a three-dimensional space. Depending on the purpose of the composite artificial intelligence device, only the objects and objects necessary for that purpose may be replaced with parts. When the purpose of the composite artificial intelligence device is not clear or when there are multiple purposes, it is preferable to reconstruct the real world with as many parts as possible.
Then, the reality corresponding virtual space unit 411 assigns an attribute to each component that is a component in the component reconstructing three-dimensional space, and the reality corresponding virtual space that treats the real world as a virtual world by a plurality of components having attributes. Generate.

Next, the self-recognition unit 412 in the virtual space arranges the three-dimensional position, posture, and movement of the object in the reality corresponding virtual space 411, and the three-dimensional arrangement of the components around the self position. And attribute relationship.
The self position can be obtained in relation to the surroundings. Further, by comparing the position, attribute, and part of the object acquired by the other artificial intelligence apparatus 100 with each other, it is possible to express itself in a three-dimensional shape in a three-dimensional space. In addition, since there are attribute information that cannot be obtained by yourself, such as the color of the object itself that can not be observed with the camera etc. provided in the object, in that case, by acquiring the latest data from surrounding artificial intelligence devices You can update your own attributes.

As described above, the real world has a space, the shape and coordinates of an object, and is reconstructed into a virtual world by parts having attributes, and a new meaning is born from the positional relationship.
Therefore, a rule for generating a meaning generated from a three-dimensional arrangement of a plurality of parts and a mutual relationship of their attributes is given in advance by the semantic rule assigning unit 413. For example, when the object is a playing card, the meaning of the arrangement differs depending on the type of game even if the card is the same playing card. Further, when the object is a vehicle, even if the distance between the two vehicles is the same, the meaning differs depending on whether the object is on the same lane or on the opposite lane.
Prepare in advance rules to produce such meanings.

The meaning generation unit 414 generates the above-described meaning from the positional relationship of itself in the reality corresponding virtual space and the relationship between these attributes.
Specifically, from the spatial positional relationship between the object itself and the surroundings, the object to which the artificial intelligence device 100 is attached has an individual unique meaning in situation determination. For example, in the case of a vehicle with the artificial intelligence device 100 attached, the vehicle has its own meaning and purpose of action, and is related by communicating with the surrounding artificial intelligence devices 100 as a whole. A comprehensive meaning is generated. The comprehensive meaning is a common meaning such as “congestion caused by an accident”.
In addition to such a comprehensive meaning, there are individual meanings possessed by the artificial intelligence apparatus 100 having individual purposes. Individual meanings are unique meanings, such as “being detoured and rushing to the destination”, and other vehicles “waiting for traffic to be resolved while eating at restaurants” .

Next, the self-situation grasping unit 415 replaces the situation placed in the three-dimensional space with the reality corresponding virtual space for understanding. The meaning generated by the meaning generation unit 415 described above has a subjective meaning in relation to itself (an object) and causes a behavior, and the self-situation grasping unit 415 can acquire a self-situation and a surrounding relational state. It is necessary to trace it over time and understand the meaning of the current situation.
Therefore, the time history simulation unit 416 confirms the self-situation that changes every time through the simulation from the past to the near future, and estimates the optimal future for the action that suits the purpose.
Then, the behavior determination unit 417 determines the next behavior of itself after understanding the above-described self-situation and surrounding situation. As a result, the behavior is uniquely determined by the individual artificial intelligence device from the comprehensive meaning and the individual meaning.
By combining a plurality of artificial intelligence devices 100 having such an action determination unit 417 in a communicable manner, a composite artificial intelligence device is realized.

Further, the learning device 418 can recreate a similar process that was previously experienced to simplify the process. Specifically, the process storage unit 418a stores all or some processes leading to the action determination of the action determination unit 417 in a part or all of the artificial intelligence devices included in the composite artificial intelligence device.
In the type classification unit 418b, the processes in the process storage unit 418a are classified and organized into similar processes and stored.

When this learning device 418 acts under the same situation, it does not recognize and determine the behavior every time, but stores the previous behavior pattern and uses the process as it is. In addition, not only in the same situation but also in similar situations, the processing method according to the case where it matches the classified situation is learned to make use of the previous experience, so the processing speed becomes faster, and as a result The reaction time of action is expedited. The more experience you have, the more effective the learning will be.
The meaning generation is generated in the learning device 418 and stored. In addition, since the arrangement and continuity of meanings produce new meanings, it is preferable that the meaning rule assigning unit 413 and the meaning generating unit 414 are provided together with the learning device 418.
In this way, a similar process that was previously experienced can be reproduced to simplify the process.

[Schematic configuration]
Next, a schematic configuration of an embodiment included in each artificial intelligence device 100 in order to realize the above basic functions and specific functions in the composite artificial intelligence device of the present invention will be described with reference to FIGS. 5 and 6. To do.
FIG. 5 is a block diagram showing a schematic configuration of one embodiment of each artificial intelligence device 100 constituting the composite artificial intelligence device of the present invention, and FIG. 6 is a block diagram showing details of the schematic configuration shown in FIG. It is.
As shown in the figure, an artificial intelligence device 100 according to an embodiment of the present invention includes an image acquisition device 501, an image feature point calculation device 502, a comparison reference signal selection device 503, a shape component generation device 504, Shape component database device 505, comparison candidate signal selection device 506, 3D map device 507 by component configuration, camera approximate position determination device 508 by GPS, comparison recognition device 509, coordinate integration device 510, and updated components The configuration includes a 3D map DB device 511, a meaning understanding determination device 512, a control device 513, and a display device (output device) 514.
Hereinafter, each part is demonstrated concretely.

The image acquisition device 501 includes a video camera unit and the like provided in each artificial intelligence device that is a unit constituting the composite artificial intelligence device, and acquires a moving image around the object to which the artificial intelligence device 100 is attached.
Specifically, the image acquisition device 501 acquires a moving image or a continuous still image.
Here, the acquired image is preferably an all-around image, and therefore, the camera provided in the image acquiring apparatus 501 is also preferably a camera that captures the entire periphery, but the configuration is not particularly limited. For example, the entire periphery may be captured by moving and rotating the camera, or the entire periphery may be covered by a plurality of cameras. In addition, a plurality of cameras may be installed to cover the entire periphery by overlapping the field of view.

As shown in FIG. 6, the image feature point calculation device 502 includes an image temporary recording unit 502a, a feature point extraction / tracking processing unit 502b, a vector calculation unit 502c, a camera vector detection unit 502d, an additional feature point extraction / A tracking processing unit 502e, a feature point coordinate calculation unit 502f, a feature point combined polygon generation unit 502g, and a feature point plane three-dimensional distribution generation unit 502h are provided.
The temporary image storage unit 502a records or primarily records the moving image acquired by the image acquisition unit 501. The image to be recorded may be in a range necessary for the calculation of the camera vector, and it is usually only necessary to have several frames to several tens of frames of video. In addition, the past video can be erased sufficiently.

The feature point extraction / tracking processing unit 502b automatically extracts a sufficiently large number of feature points or image feature parts having image features in the recorded video, and automatically features points or feature parts in the moving image. Chase. The feature point is a point shown together with the minute surrounding image, and the feature part is a minute surface having an image form.
The feature point extraction / tracking processing unit 502b automatically extracts a sufficient number of feature points necessary for calculation from the image temporarily recorded in the image primary storage unit. From the viewpoint of accuracy, it is appropriate that tens to hundreds of feature points exist in one frame.
Correspondence between the feature points of each frame is obtained by automatically tracking the feature points in each frame image. This is necessary for calculating the camera vector and for generating a feature point plane three-dimensional distribution map, which will be described later.

The vector calculation unit 502c obtains the three-dimensional coordinates of each feature point by calculation from a sufficiently large number of feature points or feature parts.
In the vector calculation unit 502c, the camera vector (camera position and rotation) corresponding to each frame image and the three-dimensional coordinates of the feature point are automatically obtained from the target feature point for calculating the camera vector by calculation of epipolar geometry.

In the camera vector detection unit 502d, a camera vector (camera three-dimensional coordinate and three-axis rotation angle is accurately obtained by statistical processing calculation from the overlapping relationship of three-dimensional coordinates of sufficiently many feature points obtained by the calculation of the vector calculation unit. ) The camera vector means a three-dimensional position and a three-axis rotation posture of the camera.
In the camera vector detection unit 502d, a plurality of triangles are formed by the feature points and the camera positions so as to improve the accuracy of the camera position, and a plurality of one camera positions are obtained, and the coordinates of one feature point are obtained. Is duplicated multiple times. The camera position is obtained by statistical processing on the assumption that the obtained feature point and the position and rotation of the camera and the position where the distribution is the narrowest range are the smallest error. When the distribution spreads over a wide range, the feature points involved in the calculation are discarded as unsuitable feature points, and the calculation is repeated. The camera vector is not always obtained alone, but is always obtained together with the feature points.
Then, by extracting the camera position and rotation as three-dimensional coordinates, the camera position, that is, the three-dimensional position and posture (corresponding to the camera position and posture) of the object itself provided with the artificial intelligence device 100 is detected. It is necessary to process the camera vector with high accuracy and in real time.

Hereinafter, a specific camera vector detection method in the camera vector detection unit 502d will be described with reference to FIG.
First, the image data acquired by the image acquisition device 501 is input to the feature point extraction / tracking processing unit 502b via the image temporary recording unit 502a (or directly), and becomes a feature point in an appropriately sampled frame image. A power point or a small area image (feature part) is automatically extracted, and a correspondence relationship between feature points is automatically obtained among a plurality of frame images.
Specifically, more than a sufficient number of feature points that are used as a reference for detecting a camera vector are obtained. An example of feature points between images and their corresponding relationships are shown in FIGS. In the figure, “+” is a feature point that is automatically extracted, and the correspondence is automatically tracked between a plurality of frame images (see correspondence points 1 to 4 shown in FIG. 9).
Here, for feature point extraction, as shown in FIG. 10, it is desirable to specify and extract a sufficiently large number of feature points in each image (see circles in FIG. 10). For example, about 100 feature points are extracted. Extract points.

Subsequently, the vector calculation unit 502c calculates the three-dimensional coordinates of the extracted feature points or feature parts, and the camera vector detection unit 502d calculates the camera vector based on the three-dimensional coordinates.
Specifically, the camera vector detection unit 502d has a sufficient number of feature positions that exist between successive frames, a position vector between moving cameras, a three-axis rotation vector of the camera, and each camera position and feature. Relative values of various three-dimensional vectors such as vectors connecting points are continuously calculated by calculation .
For example, camera motion (camera position and camera rotation) is calculated by solving an epipolar equation from the epipolar geometry of a 360-degree all-round image.

Images 1 and 2 shown in FIG. 9 are images obtained by performing Mercator expansion of 360-degree all-round images. When latitude φ and light θ are assumed, points on image 1 are (θ1, φ1) and points on image 2 are ( θ2, φ2). The spatial coordinates of each camera are z1 = (cos φ1 cos θ1, cos φ1 sin θ1, sin φ1), z2 = (cos φ2 cos θ2, cos φ2 sin θ2, sin φ2). When the camera movement vector is t and the camera rotation matrix is R, z1 ^T [t] × Rz2 = 0 is the epipolar equation.
By providing a sufficient number of feature points, t and R can be calculated as a solution by the method of least squares by linear algebra calculation. This calculation is applied to a plurality of corresponding frames.

Here, as an image used for the calculation of the camera vector, it is preferable to use a 360-degree all-round image.
The image used for the camera vector calculation may be any image in principle, but a wide-angle image such as a 360-degree all-round image shown in FIG. 9 makes it easier to select many feature points. Therefore, by using 360-degree all-round images for camera vector calculation, it is possible to increase the tracking distance of feature points and select a sufficiently large number of feature points, which is convenient for long distances, medium distances, and short distances. Feature points can be selected. In addition, when correcting the rotation vector, an arithmetic process can be easily performed by adding a polar rotation conversion process. As a result, a calculation result with higher accuracy can be obtained.
Note that FIG. 9 is a Mercator projection that uses a map projection to form a 360-degree spherical image obtained by compositing images taken by one or more cameras in order to facilitate understanding of the processing in the camera vector detection unit 502d. Although a developed image is shown, in the actual calculation of the camera vector detection unit 502d, the image does not necessarily have to be a developed image by the Mercator projection.

Next, by calculating a plurality of vectors based on each feature point using a plurality of calculation equations based on a plurality of camera positions and the number of feature points corresponding to each frame, the position of each feature point and the camera position The final vector is obtained by performing statistical processing so as to minimize the distribution of. For example, the optimal solution of the least square method is estimated by the Levenberg-Marquardt method for multiple frame camera positions, camera rotations, and multiple feature points, and the error is converged to obtain the camera position, camera rotation matrix, and feature point coordinates. .
Further, feature points having a large error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of computation at each feature point and camera position.
In this way, the position of the feature point and the camera vector can be obtained with high accuracy.

FIGS. 11 to 13 show examples of the three-dimensional coordinates of feature points and camera vectors obtained by the camera vector detection unit 502d. FIGS. 11 to 13 are explanatory diagrams visually showing a camera vector detection method, and show the relative positional relationship between a camera and an object obtained by a plurality of frame images acquired by a camera attached to a moving object. FIG.
FIG. 11 shows the three-dimensional coordinates of the feature points 1 to 4 shown in the images 1 and 2 in FIG. 9 and the camera vector that moves between the images 1 and 2.
12 and 13 show a sufficiently large number of feature points, the positions of the feature points obtained from the frame image, and the position of the moving camera. In the figure, a circle mark that continues in a straight line at the center of the graph is the camera position, and a circle mark that is located around the circle indicates the position and height of the feature point.

Here, in the calculation in the camera vector detection unit 502d, in order to obtain more accurate three-dimensional information of the feature point and the camera position, as shown in FIG. It is preferable to set feature points and to repeatedly perform a plurality of operations.
Specifically, the camera vector detection unit 502d automatically detects a feature point having a video feature in an image and uses n for camera vector calculation when obtaining a corresponding point of the feature point in each frame image. Focusing on the 2nd and n + mth frame images Fn and Fn + m, the unit calculation is repeated, and the unit calculation with n and m appropriately set is repeated.
m is the frame interval, and the feature points are classified into multiple stages according to the distance from the camera to the feature points in the image, and set so that m increases as the distance from the camera to the feature points increases. It is set so that m is smaller as the distance to is shorter. This is because the longer the distance from the camera to the feature point, the smaller the change in position between images.

Then, while sufficiently overlapping the classification of the feature points by the m value, a plurality of stages of m are set, and as n progresses continuously with the progress of the image, the calculation proceeds continuously. Then, the overlap calculation is performed a plurality of times for the same feature point in each step of n and m.
In this way, by performing unit calculation focusing on the frame images Fn and Fn + m, a precise camera vector is calculated over a long time between each frame sampled every m frames (frames are dropped). However, in m frames (minimum unit frames) between the frame images Fn and Fn + m, a simple calculation that can be performed in a short time can be performed.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames overlap with Fn and Fn + m camera vectors for which high precision calculation has been performed. Accordingly, m minimum unit frames between Fn and Fn + m are obtained by a simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by the simple calculation are Fn and Fn + m obtained by high precision calculation. The scale adjustment of m consecutive camera vectors can be made to match the camera vectors.
In this way, as n progresses continuously with the progress of the image, the scale adjustment is performed so that the error of each camera vector obtained by calculating a plurality of times for the same feature point is minimized, and integration is performed. A camera vector can be determined.
Accordingly, it is possible to speed up the arithmetic processing by combining simple arithmetic operations while obtaining a highly accurate camera vector having no error.

Here, there are various simple calculation methods depending on the accuracy. For example, (1) In the case of using a large number of feature points of 100 or more in high-precision calculation, the minimum number of simple calculation is about ten. (2) Even if the number of the same feature points is the same as the number of the feature points, if you consider the feature points and the camera position equally, there are innumerable triangles, and as many equations as there are, By reducing the number of equations, it can be simplified.
In this way, integration is performed by adjusting the scale so that the error of each feature point and camera position is minimized, distance calculation is performed, and feature points with large error distribution are deleted, and other features are added as necessary. By recalculating the points, the calculation accuracy at each feature point and camera position can be increased.

In addition, by performing high-speed simple calculation in this way, camera vector real-time processing becomes possible. Real-time processing of camera vectors is performed with the minimum number of frames that can achieve the target accuracy and the minimum number of feature points that are automatically extracted, and the approximate value of the camera vector is obtained and displayed in real time, and then the image is accumulated. Accordingly, the number of frames can be increased, the number of feature points can be increased, camera vector calculation with higher accuracy can be performed, and approximate values can be replaced with camera vector values with higher accuracy for display.
As described above, the camera vector detection unit 502d can obtain a highly accurate camera vector at high speed.

The additional feature point extraction / tracking processing unit 502e extracts an additional feature point or feature portion in the vicinity when there is a part that the user wants to know in detail regarding the three-dimensional shape or movement vector in the acquired video. Track in the video.
In the additional feature point extraction / tracking processing unit 502e, when the camera vector is obtained by calculation, the newly added feature point is easily obtained from the camera vector by simple calculation. In addition, any number of feature points that are not involved in the camera vector calculation can be added to obtain the three-dimensional coordinates.
Therefore, it is possible to arbitrarily specify the feature point extraction of the area that you want to know in detail. In the moving object extraction, the area where the moving object is expected to exist and the surrounding feature points are extracted as a group and tracked. To do.

The feature point coordinate calculation unit 502f obtains the three-dimensional coordinates and the movement vector of the additional feature point or feature part added by the additional feature point extraction / tracking processing unit 502e from the camera vector already obtained by the camera vector detection unit 502d. Obtain and output by calculation. If there is a moving body at the same time, a moving body vector signal is output.
The moving object detection may simply apply a feature point or feature part of the contradictory tracking result as a stationary coordinate system.
The feature point coordinate calculation unit 502f obtains the three-dimensional coordinates of the feature points by simple calculation using not only the feature points used for the calculation but also a sufficient number of feature points not related to the newly designated camera vector calculation. be able to. This can be calculated in a time that is orders of magnitude smaller than the calculation amount in the above-described camera vector calculation.

Therefore, it is advantageous to separate the feature point three-dimensional coordinate calculation performed simultaneously with the camera vector calculation and the determination of the feature point coordinates using the camera vector.
That is, if the camera vector is known, the three-dimensional coordinates of the feature points can be obtained in a short time by simple triangle calculation. Therefore, the feature points for camera vector calculation and the feature points for generating the three-dimensional distribution map of the feature points are not essentially distinguished and cannot be distinguished.
It is desirable to select feature points for calculation from a range that is evenly distributed over the entire image and includes a long distance, a middle distance, and a short distance in a well-balanced manner.
The feature point addition specification for generating a 3D distribution map of feature points is extracted if there is an object whose detailed shape is to be known in the image so that the object and its surroundings are densely distributed. It is desirable to specify the area.

The feature point combined polygon generation unit 502g combines the three-dimensional coordinates of the feature points obtained at the same time as the camera vector and the additional feature points obtained by the calculation with the subsequent camera vector known without any contradiction. Is generated.
In the feature point combined polygon generation unit 502g, as shown in FIGS. 15 and 16, the feature points obtained at the same time as the camera vector and the three-dimensional coordinates of the additional feature points obtained by the calculation with the later camera vector known are used. Combine unit without contradiction to generate unit polygon. There are many ways to generate a simple triangle as a unit polygon, and it is necessary to use an appropriate triangle or polygon.

When generating a unit polygon, it is effective to generate a unit polygon within a range that can be generated within the condition by setting a condition that does not exceed a certain value for the area of the polygon surface or the three-dimensional unevenness of one side.
The condition may be to measure the actual unevenness, and assuming that the brightness gradient corresponds to the unevenness, if a feature point with a large brightness gradient exists in the polygon, further subdivide to generate a unit polygon Is effective.
Also, even if the polygon is large, if there is no luminance gradient inside, it can be inferred that there is no shape gradient, and the polygon can remain large.

In this way, a polygon plane group is generated by combining a plurality of unit polygons without contradiction.
In addition to the method of generating a group of polygons by combining a plurality of small polygons, it is also effective to generate large polygons and subdivide and generate unit polygons until the luminance gradient in the polygon area is constant. .
Then, a plurality of polygon plane groups are generated in this way, and a large number of polygon plane groups obtained from a plurality of frame images are overlapped, and a polygon plane group is added to the gap and overlapped to fill the gap. A combined polygon group is generated and statistical processing is performed to generate an average polygon surface.
By repeating this polygon calculation, it is possible to continuously take out the object and the average polygon surface around it and form a three-dimensional shape of the object and its surrounding polygon. Note that the feature point distribution map can be directly guided to the comparative recognition device 509 described later without generating a Boligon.

Hereinafter, as a preferable polygon generation method for generating a polygon surface from feature points, a feature point uneven polygon division coupling generation method and a luminance gradient polygon division integration generation method will be described in detail.
[Feature Point Concavity / Concavity Polygon Split / Joint Generation Method]
When generating the unit polygon as described above, first, a standard-sized polygon is generated. If there is no feature point whose three-dimensional coordinates are known in the polygon, a new additional feature point is extracted and traced. The three-dimensional coordinates are obtained (see FIG. 17).
If there is a feature point with known three-dimensional coordinates in the polygon, pay attention to the feature point and measure the vertical distance of the known feature point included in the polygon surface or the additional feature point from the polygon surface. To do.

If a feature point whose measured value exceeds the reference value for the vertical distance is not found, leave it as it is or combine it with an adjacent polygon to generate a larger polygon, repeat this process, and within a range that does not exceed the reference value A large polygon is generated and used as a unit polygon.
If a feature point whose vertical distance exceeds the reference value is found, it is divided into polygons with new vertices, smaller polygons are generated, and work similar to that performed for standard-sized polygons for the divided polygons Apply.
Thereby, finally, the three-dimensional shape of the object can be expressed by fewer polygons.

A standard-size polygon is approximated by a surface, ignoring any three-dimensional irregularities in the polygon. Therefore, using the feature points in the standard size polygon, or by extracting additional feature points, the irregularities in the standard size polygon are examined, and the standard size is kept until it falls within the reference value. Is divided into a plurality of small polygons.
Alternatively, when the unevenness is less than the reference value, the polygons are integrated to generate a larger polygon.
By repeating this operation, the object can be expressed by the number of polygons corresponding to the unevenness, thereby enabling efficient three-dimensional shape formation.
It should be noted that the feature points and the additional feature points are different in how they are derived, and the results obtained are exactly the same.

[Luminance gradient polygon division integrated generation method]
When generating a unit polygon, first a standard size polygon is generated, and if the brightness gradient in the polygon is greater than the reference value, additional feature points are extracted and tracked inside the polygon, and their three-dimensional coordinates are obtained. The polygon is divided into smaller polygons having additional feature points as vertices inside the polygon of the size, and the division is repeated until the luminance gradient in the polygon becomes a specified value or less.
If the luminance gradient in the polygon is smaller than the reference value, adjacent polygons are combined to generate a polygon that is as large as possible within the reference value, and finally, the three-dimensional shape of the object is represented by fewer polygons.

As described above, the luminance gradient polygon division integrated generation method is a method of determining the number of polygons from the luminance gradient under the assumption that the luminance gradient is always generated where there is unevenness.
From the feature point distribution diagram, the feature points are combined to generate a polygon such as a triangle, which is converted into a polygon. However, in order to generate a unit polygon, conditions are set, and a unit polygon is generated as a large polygon because it is expected that a change in the three-dimensional shape is small when the luminance gradient in the image is small.
If the luminance gradient is the same, a condition such as limiting the three-dimensional distance between the feature points is added as a condition for connecting the feature points.

  A plurality of triangles are formed by connecting feature points within the limit distance by straight lines. In addition, since feature points exceeding the limit distance are not connected, a polygon is formed only in a certain region, and some polygons remain uncombined to form a set of partial polygons. Therefore, for areas where the distance between feature points exceeds the limit, and for areas where you want to know the detailed shape, specify the area to extract and track new feature points, and add those additional features from known camera vectors. The three-dimensional coordinates of the points are obtained, and feature points that are within the limits are combined to generate further polygons. Thereby, the partial polygons are combined to generate a wider range of polygons, and a three-dimensional shape is generated.

By adding feature points until a desired three-dimensional shape is generated, detailed polygons created by the feature points are generated, and a three-dimensional shape is generated.
Even if the feature point coordinates are obtained in the designated area, if they are not combined, even if the apparent angles are close, it means that the three-dimensional distance is far, which means that the objects are separated. In this way, the object can be separated.
Furthermore, by using the polygon plane as a feature point plane distribution signal, it is possible to create a state that is easy to compare with parts.

The feature point surface three-dimensional distribution map generation unit 502h generates the above-described camera vector signal and a moving body movement vector signal if there is a moving body, and also generates a polygon from the polygon surface generated by the feature point combined polygon generation unit 502g. A surface shape is generated and output as a feature point surface distribution signal. This is a signal indicating the primitive shape of the three-dimensional shape of the object.
If there is a moving object, a polygonal surface shape of the moving object is generated, and a moving object feature point surface distribution signal is output.

The feature point plane three-dimensional distribution map generation unit 502h generates a polygon plane group at a plurality of locations in the image.
Further, a plurality of polygon surface groups obtained from a plurality of frame images are overlapped to overlap to generate a combined polygon group, and statistical processing is performed to generate an average polygon surface.
As shown in FIG. 18, a feature point surface three-dimensional distribution map composed of polygon surfaces obtained by combining many feature points or many feature points is generated from three-dimensional coordinates of sufficiently many feature points. . Furthermore, it is possible to simultaneously draw the own three-dimensional position as the camera position and the feature point distribution in the three-dimensional space.
The three-dimensional distribution state of the feature points indicates the three-dimensional shape around the camera.

In this three-dimensional distribution diagram, the measurement accuracy can be improved by using a plurality of cameras.
For example, if the field of view overlaps, feature point tracking can be applied to images from a plurality of cameras, thus improving accuracy.
Further, if the fields of view are not overlapped, the feature points of each camera are independent, but a wide range of measurement is possible.
Furthermore, by using a plurality of cameras with different angles of view, it is possible to improve the three-dimensional positional accuracy on the wide-angle lens side and the accuracy of three-axis rotation on the narrow-angle lens side. High accuracy can be achieved.

The comparison reference signal selection device 503 outputs one or both of the feature point plane three-dimensional distribution signal and the video signal output from the image feature point calculation device 502 as a comparison reference signal.
As will be described later, the signals to be compared in the comparison recognition device 509 are the comparison reference signal and the comparison candidate signal, but the comparison reference signal selection device 503 generates the comparison reference signal.
This comparison reference signal is a part of the photographed signal generated from the photographed image by the comparison reference image selection unit. Further, it is a part of the feature point plane three-dimensional distribution map generated from the comparison reference feature point plane three-dimensional distribution section. Either or both of these are sent to the comparison recognition device and compared with the comparison candidate signal.

The shape component generation device 504 generates a reference component to be compared in the comparison recognition device 509. Specifically, a video component obtained by cutting out an object to be predicted in a moving image or a part of a characteristic part thereof, and a 3DCG component to be an object in the moving image or a part thereof are generated. A video component generator that generates a two-dimensional component, a 3DCG component generator that generates a three-dimensional component, and a surface component generator that uses only the surface shape of the three-dimensional object as a component. Yes.
In the shape component generation device 504, a plurality of components necessary for recognition, identification, understanding, and determination in a later process are generated in advance. The parts include a video part generation unit that generates from video, a 3DCG part generation unit that generates from 3DCG, and a surface part generation unit that approximates only the surface of the object.

Further, when generating a component, an attribute is given to each component. Here, the attributes are, for example, object name, object partial name, installation condition, material, weight, existence condition, shape, adjacent condition, autocorrelation coefficient, object property, price, purpose of use, manufacturing date Ri such that Ah, the color acquired by the various sensors, temperature, specific sounds, smells, etc., can be included any information related to the object.
The parts may be generated manually or automatically.
The parts are prepared in advance. For example, in the case of an object to be experienced for the first time, it may be generated in real time on the spot.

The shape component database device 505 includes a database of video components generated by the shape component generation device 504, a 3DCG component database, and a surface shape component database having a part of the surface shape of the object. The part is recorded together with the attribute and input to the comparison candidate signal selection device 506 described later.
In addition, the parts of the shape part database device 505 are configured in an arrangement close to the actual space and stored in the 3D map device 507 described later.
In the shape component database device 505, various components generated by the shape component generation device 504 are organized and stored. Since the three-dimensional space is composed of a set of a plurality of objects, a model of the predicted object is prepared in advance as a parts database.

As shown in FIG. 6, the shape component database device 505 includes a video component DB unit, a 3DCG component DB unit, and a surface shape component DB unit.
The video part DB unit stores the parts generated by the video part generation unit of the shape part generation apparatus 504 that generates parts from the video.
The 3DCG component DB unit stores components generated by the 3DCG component generation unit of the shape component generation apparatus 504 that generates components from 3DCG.
The surface shape component DB section stores the components generated by the surface component generation section of the shape component generation apparatus 504 for approximating only the surface of the object portion.

The comparison candidate signal selection device 506 designates the approximate position of itself (camera) by the camera vector signal obtained from the feature point calculation device, narrows down the location, etc., and is expected to be present in the vicinity of the approximate position Are selected from the shape part database device 505 and extracted one after another and output as a comparison candidate signal.
The comparison candidate signal selection device 506 selectively derives candidate components from the component database of the shape component database device 505. Specifically, the information on the location of the camera, for example, the approximate position obtained from the GPS, or the recognition result of the artificial intelligence system is used to determine the situation where the artificial intelligence device 100 is placed and the expected value is high. Candidate parts are selected in order and sent to the next process (comparison and recognition device 509).
Signals selected by the comparison candidate signal selection device 506 and sent to the next process include video component signals, 3DCG component signals, and planes corresponding to the component database of the shape component database device 505 as shown in FIG. The component signal and the 3D map array signal corresponding to the 3D map device 507 are obtained.

The 3D map device 507 having a component configuration is stored as a 3D map in which components stored in the database unit of the shape component database device 505 are arranged in a form close to the actual arrangement. Then, the 3D map becomes a 3D map arrangement signal input to the comparison candidate signal selection device 506 as it is.
The 3D map device 507 (or the component reconstructed three-dimensional space generation unit 410 shown in FIG. 4) according to the component configuration stores a three-dimensional shape map. A three-dimensional shape of the situation is generated from a part or all of the object image corresponding to the part in the database.

As the stored three-dimensional shape map, the following several types are conceivable.
The first is a three-dimensional map in which a three-dimensional shape is generated from a part in a database, a texture is obtained from an actual image, and an image that has been associated with the part is arranged. In this case, a three-dimensional shape map composed of parts can be considered. It is a three-dimensional shape map in which the three-dimensional shape of the surrounding situation is constituted by parts in the database.
Here, the component is a CG, or a part of a recorded real image, a map having a simple three-dimensional shape, or the like. This is a three-dimensional shape map configured by three-dimensionally arranging the parts using the parts in the database for the surrounding situation. Since it is configured as a set of parts, the attributes of the individual parts are stored and the attributes are understood.

Further, as a first special case, a three-dimensional shape map in which the three-dimensional shape of the surrounding situation is described by the part names in the database.
Although this depends on the parts in the database as described above, only the order of parts placement and the rough layout are described in text.

  The second may be a three-dimensional shape map that describes only the three-dimensional shape of the surrounding situation. This is a map in which individual objects are not recognized and the surrounding situation is expressed only as a three-dimensional shape. This is simply a map that constitutes the overall three-dimensional shape, and a map that consists of a simple surface corresponds to it. It differs from the above in that it is not partized.

  The third case is a case where the three-dimensional shape and coordinates have not yet been obtained, but the situation can be made three-dimensional at any time by associating the video image with the camera vector value. In this case, part or all of the surrounding image of the surrounding situation is drawn as images from a plurality of viewpoints. This is a map in which a three-dimensional map is constructed by using video of a characteristic part as video fragments from a plurality of viewpoints. Three-dimensional coordinates are given in advance to the video fragment.

The GPS camera approximate position determination device 508 gives the comparison candidate signal selection device 506 an approximate position for selecting a candidate component from the component database of the shape component database device 505.
The approximate camera position determination device 508 using GPS acquires approximate absolute coordinates using GPS (Global Positioning System). Since the accuracy of the GPS is limited, the GPS is used for measuring the approximate position. Then, the acquired approximate position information is sent to the comparison candidate signal selection device 506.
The comparison candidate signal selection device 506 receives the approximate position from the GPS data, sets it as the provisional position of the 3D map device based on the component structure, and sends a signal to the comparison recognition device from the plurality of parts near the provisional position. Can generate and output a 3D map array signal. As a result, a component is specified by the comparison recognition device 509 described later.

The comparison recognition device 509 compares the comparison reference signal from the comparison reference signal selection device 503 with the comparison candidate signal from the comparison candidate signal selection device 506.
The comparison reference signal from the comparison reference signal selection device 503 includes a comparison reference video signal mainly composed of an image and a comparison reference feature point plane three-dimensional distribution unit composed of a surface or a distribution of points synthesized by the feature points (see FIG. 6). .
When the comparison reference signal from the comparison reference signal selection device 503 and the comparison candidate signal from the comparison candidate signal selection device 506 do not match, the comparison recognition device 509 selects the comparison candidate signals one after another and compares them until they match. When they match, the comparison reference signal and the comparison candidate signal are associated one-to-one, thereby determining that the object has been recognized, and the comparison recognition device 509 uses the coordinates and components of the recognized comparison candidate signal. Identify and output.

As shown in FIG. 6, the comparison recognition device 509 includes a distribution shape comparison unit 509 a that compares the distribution of feature points and an object shape comparison unit 509 b that compares the three-dimensional shapes of the objects.
The comparison recognition device 509 compares the above-described feature point surface three-dimensional distribution signal and the surface component signal to obtain a three-dimensional coordinate, and if not, changes the surface component signal until they match, and finally In addition, the rough shape of the object is grasped by selecting and determining a surface component in which the feature point surface three-dimensional distribution signal and the surface component signal approximately match.
The matched part is determined by the part specifying unit together with the attribute as an object representing the object.
Next, by further narrowing down the shape, the comparison candidate signal of the 3D part or the video part is bound, compared with the surface part signal or the video signal as the comparison reference signal, and the above work is performed on the area in each frame of the moving image. Is performed continuously.

The comparison and recognition device 509 extracts and tracks feature points in the actual video corresponding to the image fragments or by matching the images with the actual video or by matching or the like. Then, the camera position is known, and the three-dimensional position of the camera can be confirmed by comparison with a surface component or directly with an object using a feature point plane distribution map obtained by vector calculation.
When comparing the feature point surface distribution map and the surface component, as shown in FIG. In addition, a comparison between the distribution of only feature points and the surface of a part is conceivable, but since the affiliation of a feature point becomes clearer after comparison with a surface part after forming a polygonal surface, the error is reduced.

The comparison recognition device 509 compares the current feature point plane three-dimensional distribution map with a part of the 3D map read from the 3D map array signal, or compares the feature point plane three-dimensional distribution chart with the surface shape component. The comparison is performed by the distribution shape comparison unit 509a.
Here, when the feature point plane three-dimensional distribution map is used as the reference comparison signal, all are performed by the distribution shape comparison unit 509a.
In addition, when the feature point plane three-dimensional distribution map is not used as the reference signal, the object shape comparison unit 509b performs all the processing.

By doing in this way, after self-position recognition by space recognition is once completed, the object shape comparison unit 509b recognizes the object. After the self-position is determined by space recognition, the correspondence between various object positions and parts on the map can be easily obtained. Once your location and surroundings are determined on the 3D shape map, you only need to check the next object, you don't have to narrow down the candidates, and you can just determine whether it is there. If there is, it is sufficient to proceed while confirming the next part as it is, or if there is not, it is judged whether only that part is lost for some reason or whether the judgment is wrong in relation to surrounding parts. If it is determined that it is no longer necessary, the part is excluded from the determination, is excluded from the map, and the action is further advanced.
If the judgment is wrong, check again with other parts and recheck the self-position and surroundings. In addition, since it may be impossible to recognize due to the shadow of another object, the self-position recognition can be completed statistically by checking with other parts.

A rough three-dimensional space around the camera is depicted in a three-dimensional distribution diagram of characteristic points formed by combining characteristic points as comparison reference signals.
Therefore, the three-dimensional space is considered to be composed of a set of a plurality of objects, and the distribution shape comparison unit 509a compares the feature point plane three-dimensional distribution map with the surface segment and predicts a rough configuration of the plane.
After that, if necessary, the object shape comparison unit 509b identifies the part of the object.

Although it is possible to compare a distribution diagram in which feature points that are not converted into polygons and a surface component are compared with each other, since the component screen is used, it is advantageous that the reference side is also a polygonal surface. In addition, if the size of the polygon is changed by the luminance gradient in the polygon, the area of the polygon surface also has information, and comparison is more advantageous and effective.
Since the model of the object to be predicted is prepared in advance in the part shape database device 505, the approximate part type and position in the three-dimensional distribution map of the feature points can generally be predicted. It is sufficiently possible to selectively select these.
Alternatively, it is possible to specify the 3D part of the object after recognizing the surface part.

In the 3D map array signal, since the parts are arranged as in reality, if the approximate position is known, candidate parts can be narrowed down to a considerable extent.
By narrowing down, the part is almost determined at the approximate position, and only the three-dimensional posture needs to be determined by the comparison device. Even if it is necessary to narrow down the candidate parts from the approximate position at the start of the system, once the camera position is determined and the parts are specified, it is sufficient to select the parts in the order in which they are arranged. Position measurement is only for the purpose of confirming the location, or it is a confirmation function when it is lost. In addition, specified parts can be used to improve the positional accuracy.
If the three-dimensional coordinates of the part are known, the camera position can be determined with higher accuracy by recognizing it.

In the distribution shape comparison unit 509a and the shape comparison device 509b, the three-dimensional shape distribution diagram created by the distribution of the video or feature points is compared with the candidate part three-dimensional shape or the already recognized surface shape. In particular, the distribution shape comparison unit recognizes only when the feature point plane three-dimensional distribution map is used as the comparison reference signal.
The current approximate position is selected from the GPS and the previous position information, and after selecting an approximate volume and size, the three-dimensional distribution of the image or feature point is compared with the three-dimensional shape of the part. The rough position of the parts can be narrowed down, for example, there is a vehicle on the ground but it cannot float in the space. Then, by comparing the two signals, the approximate position and orientation of the part are determined. If the shapes do not match, the next candidate is derived from the parts database.

In addition, it can be said that there is a high possibility that the feature point density is high where the object is present, and the feature point density is empty. From these predictions, it is possible to compare the distribution of feature points with the shape of the part.
This is a comparison between the distribution of the three-dimensional feature points and the three-dimensional shape of the part. However, at this stage, the part is not yet determined as such, but it can be said that the approximate shape and position are just the same.
As a special case, a two-dimensional distribution of feature points may be compared with a two-dimensional shape of a part. For example, this is the case when a feature point distribution on a road plane is compared with a map and a surrounding feature point distribution map is compared with a map shape.

Determine if there is a matching candidate part placement, if there is a rough placement, return to the parts database if it does not exist, and if the shape matches, make it a potential candidate for the next process To send.
If no match is found, the comparison candidate signal selection device 506 selects the next candidate component and repeats until a matching component is found, thereby identifying the component.
After that, the image of a part of the three-dimensional shape around the camera is compared with the image of the candidate part, and the candidate parts are compared one after another. This is a comparison of actual video and parts. In this case, the comparison between the three-dimensional image and the three-dimensional component is not limited. There is also a case of comparing a two-dimensional image viewed from various directions with a two-dimensional projection image of a three-dimensional component. If they match, it moves on to the next identification and fixing device.
If they do not match, the comparison candidate signal selection device 506 selects the next candidate part and repeats until a part that matches the video is found.

The coordinate integration device 510 continuously repeats the series of operations described above for all the frames of the moving image, adds the coordinates of the corresponding part and the movement vector to the moving object, and in a stationary object, Integrate the coordinates of multiple recognized parts.
In the coordinate integration device 510, a part in the case where the two-dimensional projection plane of the image matches the part is selected and specified, and the three-dimensional position of the part when matched is fixed as the three-dimensional coordinate of the specified part.

Since the self-position of the artificial intelligence apparatus 100 has already been obtained as a camera vector as described above, the camera position coordinates and component coordinates are dropped into the reference three-dimensional coordinates and integrated as coordinates. Then, the process of identifying and fixing the target part from the candidate parts by matching the shape is repeated.
If a corresponding part is not prepared, there is no matching part, so such a case is also assumed, in which case only a rough shape from the feature point distribution is recognized, and It is made to register as an undetermined object without knowing which part corresponds to.

In the self-confirming unit of the coordinate integrating device 510, there is no part to be matched among the plurality of parts, but it is possible to prepare its own three-dimensional shape in advance. Since the self-position is known by the camera vector detection device 502d, it is possible to arrange its own three-dimensional shape at the coordinates.
This is, for example, a vehicle in the case of an automatic driving system, and the shape of a robot in the case of an artificial intelligence system.

The moving part coordinate tracking unit (moving body confirmation unit) of the coordinate integration device 510 confirms the parts of the moving body and integrates them as a moving body in the coordinate system.
The feature point plane three-dimensional distribution map can classify the moving object separately from the stationary object. As shown in FIG. 20, by generating a feature point plane three-dimensional distribution map belonging to a moving object, a common camera vector exists in a group of feature points belonging to one moving object. However, the moving body and the camera position are relative values and do not yet have absolute values.
Therefore, the camera position of the moving object is determined by substituting the camera position obtained with respect to the known stationary coordinate system into the moving object camera position, and as a result, the moving object position is determined. Even if the part cannot be specified, the rough shape can be integrated and imported into the coordinate system by approximating the rough shape as it is or by approximating it with the surface part from the 3D distribution map of the feature points belonging to the moving object. .

The 3D map DB device 511 having the updated component configuration reconfigures the above-described three-dimensional shape around the camera with a plurality of components until it meets the purpose.
Thereby, a three-dimensional space by a plurality of parts is regenerated. The multiple parts all have names and attributes, and the space expressed as a combination of them is reconstructed.
If the system has a 3D map device with a component configuration in advance, this updated 3D map DB device 511 means that the map has been updated to the latest data. This means that the system has newly created a 3D map device with a component configuration in advance.

By exchanging information between the plurality of artificial intelligence devices 100 in the above process steps, the accuracy of component identification is improved and the accuracy of coordinate integration is improved. In the display device 514, all directions, all The object can be displayed from the area.
Further, this makes it possible to improve meaning understanding in the meaning understanding determination device 512.

The meaning comprehension judging device 512 is a three-dimensional part of the part by the 3D map DB device 511 (or the part reconstructing three-dimensional space generation unit 410 shown in FIG. 4) based on the part updated by the coordinate integration apparatus 510. Understand the situation and meaning of yourself (camera) from the reconstructed status, and recognize the properties of the object alone from the information given in advance as the attributes of the parts of the object, and multiple recognition targets each with a genus Understand the meaning of the arrangement of objects, or understand the meaning of the situation where they are placed from their composite information, and judge the next action.
Then, as shown in FIG. 20, the control device 513 is controlled by the output of the meaning understanding determination device 512, and automatically controls and moves itself (camera) or the target work device (robot vehicle or the like). Can move, control and take action. Furthermore, by exchanging information with other artificial intelligence devices 100 at the above-mentioned process stage, accuracy is improved, and comprehensive judgment that is difficult with a single artificial intelligence device is made, and judgments that are consistent with the overall judgment are made individually. Can control and take action.

The meaning understanding determination device 512 recognizes the three-dimensional space again as a set of a plurality of parts whose shapes, coordinates, and attributes are known based on the attributes of the reconstructed parts, and individually understands the three-dimensional space. To understand as a collection of objects. The space is understood not only as the position coordinates of the object but also as a collection of knowledge such as names, properties, and backgrounds that are understood in advance as its attributes.
The attributes of a part include the object name, part name of the object, installation conditions, properties, material, weight, color, temperature, existence conditions, shape, adjacent conditions, autocorrelation coefficient, etc. Say no information.
Further, the meaning understanding determination device 512 can understand the current meaning of the target work, and can determine whether the site is appropriate for the target work and determine future actions (for example, in which direction to proceed). For example, there are various understandings of the determination in line with the work for the purpose, such as understanding safety, understanding danger, determining that it is possible to proceed straight ahead, and determining stoppage. Specifically, in the recognized surrounding situation, it is safe if there are no obstacles in the traveling direction of itself, and it is judged dangerous if there are obstacles.

The control device 513 operates and controls the target device in accordance with the purpose based on the judgment of the meaning understanding device 512.
For example, as shown in FIG. 20B, in the case of a vehicle, a steering wheel, an accelerator, a brake, a direction indicator, a horn, etc. are judged by the meaning understanding device, automatically operated, and controlled. Automatic control of equipment is possible.
In addition, the contents of control can be displayed, and humans can understand and work on it.
As shown in FIG. 20B, the target automatic control can be performed by appropriately controlling each control item.

The display device (output device) 514 reconstructs all or part of the moving image with known parts and, if necessary, the position (three-dimensional) of itself (camera) obtained as a camera vector therein. The coordinate position) and posture (three-axis rotation posture) are displayed on a monitor screen or the like, or output by voice or the like.
The display device 514 can display surrounding conditions as viewed from the position of the camera itself.
Further, it can be output as a voice guide via a speaker or the like.
The contents of the voice guide include, for example, the contents listed in FIG.

[Position / orientation calculation based on reference image]
Next, an embodiment for obtaining the position and orientation of an object using a reference image in the composite artificial intelligence apparatus of the present invention as described above will be described with reference to FIGS.
FIG. 22 is a block diagram showing reference data of a reference image for reinforcing the position and orientation of an object in the composite artificial intelligence apparatus of the present invention. FIG. 23 shows a position and orientation calculation means using the reference image shown in FIG. It is a block diagram which shows schematic structure of these.
As shown in these figures, the reinforcing means for the composite artificial intelligence apparatus captures in advance a reference image of a movement path or movement range of an object that is predicted in advance, and camera vector calculation corresponding to each frame image of the reference image. The data is stored in advance, and further, if necessary, the reference video is converted into the three-dimensional data by converting the reference video with the camera vector calculation data, and the correspondence between the reference video and the moving image is obtained. The GPS data and the like are reinforced to improve the accuracy of the camera position and orientation.

When the object moves, it is particularly important to obtain the position of the object with high accuracy. For this reason, in the composite artificial intelligence apparatus of the present invention, the GPS data as described above may not be sufficient as information about the position and orientation between objects.
Therefore, in this embodiment, a reference image with a high position and orientation accuracy is prepared in advance as a video storage unit.
As shown in FIG. 22, in the process (1-8 shown in FIG. 22) for obtaining a camera vector of a moving image from the reference video and its camera vector, the reference image is processed and referenced to which stage of processes 1-8. Whether it is stored as data is arbitrary. The final process is to obtain up to three-dimensional data of the video, but the level of the process may be determined from problems such as cost and labor.

In other words, as far as the reference data of the reference video to be stored and held, how much of the above-mentioned eight types of data is free in principle is merely a matter of cost and calculation time.
Therefore, as the range to be stored as the reference data, for example, a part of the standard image can be stored in three dimensions, or only the standard video that is not processed at all can be stored. Although it is appropriate to perform the camera vector calculation from the reference image as a pre-processing, it is also possible to obtain the calculation from the reference image on the spot.

Hereinafter, the control of the position and orientation of the object using the reference image will be described with reference to FIG.
As shown in FIG. 23, first, in order to obtain the camera vector of the reference data and the acquired data, a flow of feature point extraction unit → feature part extraction unit → feature surface extraction unit → tracking unit → camera vector calculation unit →. It becomes. Here, a case where the camera vector is obtained and recorded on a recording medium will be described.
First, the position and orientation data of the standard video as highly accurate reference data is obtained. Then, the acquired video is processed in real time to obtain a camera vector.

As shown in FIG. 24, the reference data is a standard video camera vector, the acquired data is a video video camera vector, a known standard video camera vector is incorporated into a video video camera vector calculation, and a series of camera vectors is obtained. And the known camera vector data is substituted to determine the camera vector of the moving image as the entire camera vector.
In addition, by comparing the camera vector of the known reference image with the camera vector of the moving image image obtained by calculation on the spot, the relationship between the two is obtained from the correspondence of the characteristic part of the image, and the two camera vectors are correlated. The position of the camera itself attached to the artificial intelligence device can be determined.
In this way, the position of the object can be obtained with high accuracy as the deviation of the three-dimensional position and orientation from the reference image.

In that case, the CV calculation of the reference image is performed on the spot as needed.
Although it may be calculated as the deviation of the camera vector data from the reference video as described above over all the frames of the video image, in order to shorten the calculation time, the above-mentioned calculation is performed with a sufficient frame interval of the reference image. The intermediate frame can be simply interpolated as shown in FIG. 25 by simply performing camera vector calculation on the moving image side.
By doing so, the number of frames of the reference image can be reduced, so that the amount of data can be reduced and the calculation processing time can be greatly shortened, which is preferable.

Furthermore, as another embodiment, instead of the above-described reference image, a plurality of feature portions suitable for obtaining correspondence with the moving image in the reference image are selected in advance, cut out, and held as a reference feature portion image. Thus, it is possible to reinforce the GPS data and improve the accuracy of the camera position and orientation. In this case, if necessary, the reference feature part image and the camera vector calculation data can be converted into two-dimensional data or three-dimensional data, and the correspondence with the acquired moving image can be obtained.
Since the amount of data is large in the above-described reference image, the amount of data can be reduced by selectively having, as a partial image, only a characteristic portion necessary for camera vector calculation in the reference image. This characteristic part data is recorded on a recording medium in the same manner as in the case of the reference image.

[High-precision three-dimensional arithmetic unit]
Next, with reference to FIG. 26, an embodiment in which the composite artificial intelligence apparatus of the present invention is further provided with a high-precision three-dimensional arithmetic device will be described.
FIG. 26 is a block diagram illustrating a schematic configuration of an embodiment in which the high-accuracy three-dimensional arithmetic device 515 is provided in the artificial intelligence apparatus illustrated in FIG.
A high-precision three-dimensional calculation device 515 shown in the figure includes a three-dimensional shape tracking unit and a high-precision camera vector calculation unit.
The three-dimensional shape tracking unit positions the camera vector obtained by the image feature point calculation device 502 as a rough camera vector, and uses a part of the three-dimensional shape of the data having the three-dimensional information generated in the subsequent process to obtain a third order. Automatic tracking of the original shape.
A camera vector is output with high accuracy from the obtained tracking data by a high-precision camera vector calculation unit.

As described above, the feature point extraction / tracking processing unit 502b automatically tracks feature points in a plurality of inter-frame images, but the number of feature point tracking frames is limited due to disappearance of feature points. Sometimes. In addition, since the image is two-dimensional and the shape changes during tracking, the tracking accuracy has a certain limit.
Therefore, the camera vector obtained by this feature point tracking is positioned as an approximate value, and information on the three-dimensional shape obtained in the subsequent process is traced on each frame image, and a high-precision camera vector can be obtained from the trajectory. .
Tracking of 3D shapes is easy to match and correlate, and 3D shapes can be tracked over many frame images because the 3D shape does not change depending on the frame image, which makes it possible to calculate camera vectors. Accuracy can be improved. This is possible because the approximate camera vector is known and the three-dimensional shape is already known.

Even if the camera vector is an approximate value, the error of the 3D coordinates over multiple frames is an approximate value, and the error of the 3D shape of a part of the image is small, so it is possible to compare and track the 3D shape sufficiently. .
Here, the three-dimensional shape data to be tracked include the following.
(1) Feature point 3D distribution shape output from the comparison reference feature point surface 3D distribution
(2) Surface-shaped components output from the surface-shaped component identification unit
(3) Parts output from the part specifying unit Since these parts each have a three-dimensional shape, they are subject to tracking by the high-precision three-dimensional shape calculation device 515.

From the above, the three-dimensional shape tracking unit tracks the three-dimensional shape, and the above-described (1), (2), (3) and combinations thereof are conceivable.
Furthermore, a three-dimensional shape can be converted from a camera position into a two-dimensional image and tracked as a two-dimensional image.
Since the approximate value of the camera vector is known, projection conversion can be performed on a two-dimensional image from the camera viewpoint, and it is also possible to follow a change in the shape of the object due to movement of the camera viewpoint.

[Multiple image acquisition device]
Next, in the composite artificial intelligence device of the present invention, an embodiment in which each artificial intelligence device includes a plurality of image acquisition devices and a plurality of image feature point calculation devices corresponding thereto will be described with reference to FIGS. .
As shown in FIGS. 27 and 28, the present embodiment has a configuration substantially similar to that of the artificial intelligence apparatus 100 shown in FIGS.
In the image acquisition device 501, a plurality of moving images are acquired by a plurality of synchronized video cameras.
The image feature point calculation device 502 has the same number of function units as a plurality of video cameras, and outputs corresponding to the number of cameras calculated corresponding to each camera belong to each camera. In the same number of image feature point calculation devices as the number of cameras, the calculation is performed while relating the same function units.
As a result, the calculation accuracy can be improved, and a more accurate camera vector, feature point three-dimensional coordinates, and a wide range of feature point plane three-dimensional distribution map can be generated.

The details of performing high-precision camera vector calculation using a plurality of camera images obtained by the plurality of image acquisition devices 501 will be described below with reference to FIGS.
When using multiple camera images in camera vector calculation, the positional relationship between a wide-angle camera with a wide field of view and a narrow-angle camera with a narrower field of view than this wide-angle camera is fixed and used at the same time. By inputting the wide-angle field of view image and the narrow-angle field-of-view image from the narrow-angle field-of-view camera respectively, and substituting the camera vector value obtained from the calculation from the wide-angle field-of-view image when calculating the camera vector using the narrow-angle field of view, An accurate camera vector can be obtained.

In general, since a wide-angle lens has a wide field of view, the tracking point can be taken over a long distance, so that the positional accuracy is high. However, since the number of pixels per field angle is small, the rotation accuracy is lower than that of a narrow angle lens.
On the other hand, since the narrow-angle lens has a narrow field of view, the positional accuracy cannot be obtained. However, since the number of pixels per angle of view is large, the angle accuracy is high.
Therefore, it is possible to obtain a highly accurate camera vector by performing camera vector calculation using both a wide-angle field image obtained by a wide-angle lens and a narrow-angle field image obtained by a narrow-angle lens.

The camera vector calculation using images obtained by the wide-angle lens camera and the narrow-angle lens camera will be described in more detail.
When using a wide-angle field-of-view camera and a narrow-angle field-of-view camera, both cameras are fixedly installed in a vehicle or the like and used in synchronization.
The video output from the wide-angle viewing camera is image-compressed by a method such as AVI, and is temporarily stored in a recording unit (image temporary recording unit). Then, necessary image frames are called and camera vector calculation is performed.

On the other hand, the camera vector calculation is also performed on the narrow-angle field camera side based on the acquired image, but the accuracy of the three-dimensional position coordinates is low, so the camera vector obtained from the wide-angle field image of the wide-angle field camera The camera vector is calculated by substituting the three-dimensional coordinates of the values into the three-dimensional coordinates obtained from the narrow-angle field image of the narrow-angle field camera.
Since the camera vector of the narrow-angle field camera obtained in this way has a sufficiently large number of pixels per field angle compared to the wide-angle field-of-view camera, a highly accurate coordinate axis rotation angle can be obtained.
If necessary, the position accuracy can be further improved by substituting the high-precision coordinate axis rotation angle obtained here as the coordinate rotation angle on the wide-angle viewing camera side.

The wide-angle lens camera and the narrow-angle lens camera are integrally arranged by being fixed so that the positional relationship does not change. Wide-angle and narrow-angle cameras are driven synchronously.
The video output of each camera is recorded in a recording unit. In the case of image stabilization as post-processing, the video output is compressed by AVI or the like, and all frames are recorded in the storage. In addition, when image stabilization is performed as camera attitude control in real time, or for the purpose of position control and attitude control of the camera holding mechanism and the vehicle itself, it is necessary for real-time processing rather than recording images of all frames. It suffices to record only the number of frames. However, if the video itself is required, all necessary images are recorded.

The video output of the wide-angle lens camera and the narrow-angle lens camera is temporarily stored and the camera vector is calculated, but the two camera vectors are basically the same, simply translated, or rotated by a constant number There is a relationship, and if you ask for one, you ask for the other.
Here, since the camera vector obtained by the wide-angle lens camera has a wide field of view, the feature points can be tracked over a long distance, and the three-dimensional position accuracy can be high. On the other hand, the video output of a narrow-angle lens camera has a narrow field of view and the tracking distance of feature points is shortened, so that the three-dimensional position accuracy cannot be obtained, but since the angle of view per pixel is small, the rotation accuracy can be high.

Therefore, by combining the advantages of both cameras, the three-dimensional position coordinates are obtained by calculation on the wide-angle lens side, and the obtained three-dimensional position coordinates are adopted as the three-dimensional position coordinates of the narrow-angle lens camera. By obtaining the camera vector calculation at, a highly accurate three-axis rotation signal can be obtained.
Furthermore, if necessary, it is possible to calculate a more accurate three-dimensional position by substituting the three-axis rotation signal as the three-axis rotation angle on the wide-angle lens camera side. By repeating this, the accuracy can be further improved.

As described above, after a highly accurate three-dimensional position and three-axis rotation are obtained by real-time processing, the three-dimensional position and posture of a camera holding mechanism, a vehicle carrying the camera, etc. are controlled using this as a posture control signal. Can do. However, in the case of real-time processing, there is no need for AVI image conversion for the purpose of camera vector calculation.
In addition, for the actual calculation of image stabilization of the recorded image, the signal from the shake component signal is processed, but the image pasted on the spherical surface is rotated by the shake component signal as in the case of handling 360 ° video. This can be realized by a correction method, or by converting a normal flat image that is not a spherical surface like a normal image into a plane so that it is rotated in accordance with the swing angle and flattened. Actually, either a spherical transformation or a planar transformation is possible for a planar image or a spherical image.

FIG. 29 shows the relationship between a wide-angle camera and a narrow-angle camera.
As shown in the figure, each camera performs feature point extraction and feature point tracking in each field of view, so the fields of view do not necessarily have to overlap.
It is only necessary that the positional relationship of each camera is fixed.
Since the position coordinates of the camera vector obtained by one camera are relative values, it is necessary to perform some scale calibration later.However, if the same feature point can be tracked by overlapping the field of view of the camera, the parallax can be reduced. Since measurement is possible, the absolute distance can be obtained, which is advantageous for measurement and accuracy is further improved.
FIG. 30 shows a case where the fields of view of the cameras are overlapped.

As shown in FIG. 30, when overlapping the visual field of a camera, it is preferable to install a dedicated visual field overlapping camera. As shown in the figure, the three-dimensional position accuracy is improved by arranging the narrow-angle field-of-view cameras so that they overlap in the expected traveling direction of a moving body such as a vehicle and extending the tracking path of feature points. Can be made.
Further, the narrow-angle field-of-view camera of the horizontal independent field shown in the figure is set so as to have high sensitivity at the three-axis rotation angle.
When such a field-of-view overlap camera is used for image stabilization means (see an image stabilization unit 604 shown in FIG. 33 to be described later), the final output triaxial rotation vector of the image feature point calculation device 502 is used as a correction signal. To stabilize the image.
Note that the fields of view of the wide-angle lens and the narrow-angle lens do not necessarily overlap.

  Further, when obtaining a highly accurate camera vector, as shown in FIG. 31, a plurality of narrow-angle cameras are used simultaneously with a wide-angle camera, as shown in FIG. In this case, the field of view is overlapped, and the entire field of view is grasped with a wide-angle field-of-view camera. By tracking feature points in the direction, it is possible to realize high-precision camera vector calculation for position and orientation. The narrow-angle field-of-view cameras at the left and right ends shown in FIG. 31 are installed so as to be highly sensitive to the three-axis rotation angle.

Further, as shown in FIG. 32, a plurality of wide-angle visual field cameras that grasp the entire visual field can be provided.
The example shown in FIG. 32 is a case where a plurality of (two) wide-angle field-of-view cameras are installed with overlapping fields of view when the distance from the ground is approaching, such as when an aircraft is taking off and landing.
At the time of takeoff and landing of the aircraft, the ground speed increases as it approaches the runway, so that it is difficult to detect and track feature points, and a wide-angle visual field camera is installed laterally outward as shown in FIG.
In addition, since the absolute distance can be measured if there is an overlapping field of view as described above, the relative distance obtained by the camera vector calculation can be calibrated and converted into the absolute distance.

Next, with reference to FIGS. 33 to 37, an embodiment for performing mobile object detection / stationary coordinate system separation / moving body coordinate system separation in the composite artificial intelligence apparatus of the present invention will be described.
FIG. 33 and FIG. 34 are block diagrams showing a schematic configuration of an artificial intelligence apparatus that performs mobile body detection, stationary coordinate system separation, and mobile body coordinate system separation according to the present embodiment.
In the figure, a moving image acquisition unit 601 acquires a moving image from a camera attached to an object provided with the artificial intelligence apparatus 100.
The feature point detection / tracking unit 602 extracts a sufficiently large number of feature points of a stationary object in the moving image obtained by the moving image acquisition unit 601, and extracts a sufficient number of extracted feature points over a plurality of frame images. Chase.
The camera vector calculation unit 603 calculates a camera vector (camera position and orientation) from the tracking result of the feature points.

The image stabilizing unit 604 stabilizes the image by stopping the shaking due to the rotation of the image from the camera vector.
The stationary system extended intersection extraction unit 605 rearranges the tracking locus of the feature points on the image stabilized by the image stabilization unit 604, and extends the tracking locus of a plurality of feature points on the stabilized image. Find a pair of extended intersections where the lines meet in a stationary coordinate system.
The divided image classifying unit 606 divides the stabilized image into divided images of small regions, obtains corresponding points over a plurality of frame images for each divided image, and determines whether each divided image passes through the extended intersection. Depending on whether or not it is classified as a stationary coordinate system or other.

The stationary coordinate system classification extraction unit 607 extracts and temporarily records the divided images determined as the stationary coordinate system by the divided image classification unit based on the extended intersection.
The classification extraction unit 608 other than the stationary coordinate system extracts and temporarily records the divided images that are determined to be other than the stationary coordinate system by the image classification unit depending on whether they belong to the extended intersection.
The static coordinate system three-dimensionalization unit 609 obtains a corresponding point on the extension connecting the extended intersection point and the center of the divided image with respect to the divided image determined as the static coordinate system obtained by the static coordinate system classification extracting unit 607. Obtain the three-dimensional coordinates of the divided image.
The moving body divided image tracking unit 610 tracks the moving body divided image separated by the classification extraction unit 608 other than the stationary coordinate system over a plurality of frames.

The moving object divided intersection classification unit 611 obtains the tracking extended intersection of the moving object divided image from the tracking trajectory by the moving object divided image tracking unit, and classifies the moving object divided image by moving object for each obtained moving object extended intersection. To do.
The moving object-specific three-dimensionalization unit 612 obtains the three-dimensional coordinates of the moving object-specific divided image from the moving object-specific divided images classified by the moving object by the moving object extension intersection classifying unit and the tracking result thereof, thereby moving the moving object. Find the shape and its coordinates.
The scale matching unit 613 shares the scale of each moving body with the stationary coordinate system based on a common known length obtained by the moving body three-dimensionalization unit or an object common to a plurality of cameras.
The coordinate integration unit 614 integrates the three-dimensional coordinates for each moving body and the coordinates of the stationary coordinate system that have been scaled by the scale matching unit.

The image dividing unit 615 divides the stabilized image into divided images of small areas.
In the block image divided by the image dividing unit 615, the rubber string correlation unit 616 combines the block images so that the arrangement order is not changed as if the block images are connected by rubber strings, and the blocks are expanded and contracted. If there is a large luminance difference between them, it will be impacted and loosely coupled so that the rubber string will be cut, and correlation or matching between frames will be taken. As a result, even if some correlations or matchings are incomplete, the entire image can be prevented from being lost by arranging the blocks without changing their order.

According to the artificial intelligence apparatus 100 of the present embodiment configured as described above, moving object detection, stationary coordinate system separation, and moving object coordinate system separation are performed as follows.
First, the moving image acquisition unit 601 acquires a moving image obtained by a camera attached to an object as a partial function of artificial intelligence. If the image is distorted, correct the image and proceed to the next process.
The feature point detection / tracking unit 602 detects feature points over a plurality of frames and tracks them.
As with the camera vector detection unit 502d shown in FIG. 6, the camera vector calculation unit 603 calculates a camera vector from the tracking trace of the feature points. As described above, the camera vector is a vector expressing the position and orientation of the camera with six degrees of freedom (X, Y, Z, Φx, Φy, Φz).
The image stabilization unit 604 stabilizes the image by a process of feature point extraction / tracking → camera vector calculation → rotation correction.

The stationary system extended intersection extraction unit 605 obtains a pair of extended intersections where the extension lines of the tracking trajectories of a plurality of feature points intersect in the stationary coordinate system on the stabilized image.
On the stabilized plane conversion image (normal perspective image), the tracking trajectory of the feature point and the divided image is a straight line, and a pair of extended intersections is obtained as the intersection of the straight lines. If tracking is performed along the flow of time, the spring point in the traveling direction and the vanishing point (different from the vanishing point in perspective) on the opposite side of the traveling direction are obtained as extended intersections. If the time flow is traced back, the relationship will be reversed, but a pair of extended intersections will always be found.
In addition, when a plurality of feature points and divided images are tracked on a stabilized spherical image, the intersection of the trajectories becomes a great circle, and a pair of extended intersections are obtained as two points where the great circle intersects on the spherical image.

The divided image classifying unit 606 divides the stabilized image into divided images of small regions, obtains corresponding points over a plurality of frame images for each divided image, and determines whether each divided image passes through the extended intersection point. Depending on whether or not it is classified as a stationary coordinate system or other.
It is preferable to use a rubber string correlation method that can obtain a good correspondence for each frame by obtaining the correspondence with priority given to the arrangement of the divided images.
The rubber string correlation method is to block images and combine each block image so that the arrangement order is not changed, as if the vertical and horizontal lines are connected in a grid pattern with rubber strings, and so as to expand and contract. If there is, the rubber string is loosely joined so that it becomes an impact and the correlation or matching between frames is taken, and even if the correlation or matching is incomplete, the order of the blocks is not changed It is the same method as the rubber string correlation theory that the whole image is not lost by interpolating with.

In the stationary coordinate system classification extraction unit 607, there is always an extended intersection point in the stabilized image, the extended intersection point is a fixed position with respect to the image frame, and the tracking trajectory of the divided image or the feature point tracking trajectory is a flat image. Using the property of a straight line originating from the extended intersection, a divided image determined as a stationary coordinate system by the divided image classification unit based on the extended intersection is extracted and temporarily recorded.
Specifically, as shown in FIG. 35, a corresponding point is obtained on a straight line connecting the extended intersection and the divided image, and if the corresponding point is obtained, it is classified as being present in a stationary coordinate system as a tracking point. To do.
By doing in this way, a corresponding point can be calculated | required in a very short time compared with the corresponding point search in a surface, without calculating | requiring the corresponding point in a surface.

What exists in the stationary coordinate system is a straight line in the case of a plane image and a great circle in the case of a spherical image when connecting the extended intersection and the divided image or feature point.
At this time, in the tracking process for finding the corresponding point on the straight line connecting the extended intersection and the divided image or the feature point, the deviation from the straight line is corrected by searching for the corresponding point to a range slightly beyond the straight line. By recalculating the extended intersection point, a more accurate extended intersection point is obtained. A highly accurate camera vector calculation can be performed from an extended intersection with high accuracy.
In general, the rubber string correlation theory is the correspondence of the surface, but if you combine the extended intersection method and the rubber string correlation theory, you can make a one-dimensional rubber string that finds the corresponding point on the extended straight line from the extended intersection point. The corresponding point calculation time is considerably shortened and the accuracy is improved.
Further, the divided image and the feature point have the same meaning with respect to the extended intersection and are not distinguished. Therefore, whether an extended intersection point is obtained by tracking feature points or a feature point is obtained from divided images has the same physical meaning.

The non-stationary coordinate system classification extraction unit 608 extracts and temporarily records the divided images determined by the image classification unit based on whether or not they belong to the extended intersection point.
The stationary coordinate system three-dimensionalization unit 609 obtains a corresponding point on the extension connecting the extended intersection and the center of the divided image for the divided image determined as the stationary coordinate system obtained by the stationary coordinate system classification extracting unit 607, Obtain the three-dimensional coordinates of the divided image.
On the other hand, the moving object divided image tracking unit 610 tracks the moving object divided image separated by the classification extraction unit 608 other than the stationary coordinate system over a plurality of frames.
In the extended intersection classification unit 611 for each moving object, as shown in FIG. 31, the tracking extension intersection of the moving object divided image is obtained from the tracking trajectory by the moving object divided image tracking unit. The divided images are classified by moving object.

The moving object-specific three-dimensionalization unit 612 calculates the three-dimensional coordinates of the moving object-specific divided image from the moving object-specific divided images classified by the moving object by the moving object extended intersection classifying unit 611 and the tracking result thereof. The scale matching unit 613 obtains the body shape and its coordinates. The scale of each moving body is stationary based on a common known length obtained by the moving body three-dimensionalization unit or an object common to a plurality of cameras. Share with coordinate system.
The coordinate integration unit 614 integrates the three-dimensional coordinates for each moving body and the coordinates of the stationary coordinate system that have been scaled by the scale matching unit. In the case of multiple cameras, the absolute length is calculated if the distance between the cameras is fixed, but if there is only one camera, it will be a relative value, but it can be obtained by obtaining information from other artificial intelligence devices. The length is calculated.
In addition, a moving body-specific area large classification unit (not shown) is provided in front of the classification extraction unit 608 other than the stationary coordinate system in advance, and the group of divided images obtained from the classification extraction unit 608 other than the stationary coordinate system is largely classified. In addition, the moving object divided image can be tracked.

As a unit to be divided in the divided image unit 615, for example, about 7 * 7 to 15 * 15 pixels is desirable.
In the stationary coordinate system three-dimensionalization unit 609 and the three-dimensionalization unit for each moving object 612, in principle, all the divided images are tracked over a plurality of frames, but instead of searching for the corresponding points in the dark clouds, You only have to search for the extension from the extended intersection. This enables accurate tracking.
In addition, the component database unit 620 is used by the comparison unit 621 when the outline of the three-dimensional shape is acquired by the three-dimensional moving unit 612 or when the absolute scale of the three-dimensional shape is acquired by the scale matching unit 613. In comparison with the three-dimensional part (see FIG. 34), recognition is performed for each moving body (moving body recognition unit 619), and further replacement with the part can be performed (part replacement unit 622).
In this way, by integrating coordinates of the replaced part with the stationary coordinate system, the moving object can be expressed in 3DCG in the stationary coordinate system.

Further, in the present embodiment, in order to perform coordinate determination for each moving object and integration into a stationary coordinate system, a coordinate system determining unit 617 for each moving object can be provided as shown in FIG.
The moving object-specific coordinate system determination unit 617 detects a virtual moving object-specific camera vector from the moving object-specific divided image tracking trajectory obtained by the moving object-specific extended intersection classification unit 611, and detects the detected virtual moving object-specific camera vector. The coordinate system of the plurality of virtual camera vectors for each moving body is coordinate-transformed so as to match the camera vector of a known stationary coordinate system. Thereby, the coordinate system for each moving object can be transformed into the stationary coordinate system, and the moving object can be integrated into the stationary coordinate system for each moving object.

The movement of the moving body and the camera is always a relative movement, and it cannot be distinguished whether the camera moves or the object moves, which means the same thing. The camera vector at this time is defined as a virtual camera vector.
As shown in FIGS. 36 and 37, the virtual camera vector exists inherently in the moving object. That is, feature points of a plurality of moving objects belong to unique virtual camera vectors.
Divided images classified by moving object (divided image by moving object) move with respect to the camera, but generally the camera is also moving, so the camera and its movement are first expressed in a stationary coordinate system by camera vector calculation. To do.
In addition, since the moving body viewed from the camera has a temporary stationary coordinate system, if a temporary camera vector for each moving body is directly calculated, a temporary camera vector with the moving body as a stationary coordinate system can be obtained. As a result, a coordinate system for each moving body is obtained, and the coordinate system can be determined for each moving body.

The above operations are performed in the moving object-specific coordinate system determination unit 617.
When there is one camera, since the cameras are the same, the coordinate of the divided image for each moving object is expressed in the stationary coordinate system by matching the camera vector with the temporary camera vector.
In addition, since the data of the divided images for each moving object is generally smaller than the divided images of the stationary coordinate system, the three-dimensional shape of the moving object can be obtained by combining it with the relative three-dimensional data of the same moving object received from other artificial intelligence devices. It becomes easy to obtain coordinates. Also, the correct absolute scale can be obtained by integrating data from artificial intelligence devices in multiple directions.
In general, the 3D data obtained with one camera is a relative value, and in order to convert it to an absolute value, it is converted based on a known length in the image, or information from other cameras is acquired. Then, it can be converted into an absolute scale based on a known distance such as a distance between cameras.

Furthermore, in the present embodiment, as shown in FIG. 34, it is possible to provide a moving body-specific lock-on unit 613 that locks on the moving body in the movement-specific coordinate system.
The moving-unit-specific lock-on unit 613 uses a moving-unit-specific coordinate determination unit that obtains a moving-unit-specific coordinate system based on the moving-unit-specific extended intersection obtained by the moving-unit-specific extended intersection classification unit 611 or the camera vector. Determine the coordinate system of the moving object and fix the extended intersection of any moving object, or control the camera direction based on any position and orientation defined in relation to the coordinate system for each moving object, or The image is stabilized and displayed in the coordinate system of an arbitrary moving body without controlling the camera.

As described above, the moving object-specific extended intersection classification unit 611 and the moving object-specific coordinate system determination unit 617 can generate a coordinate system for each moving object.
Therefore, the moving body can be displayed (locked on) like a stationary coordinate system by controlling the camera direction and always tracking a specific moving body coordinate system.
In addition, a certain coordinate system is always tracked from the acquired video by image processing without controlling the camera, and the moving object coordinate system is treated as if it were a stationary coordinate system, and displayed, that is, locked on. be able to.

In the present embodiment, the image stabilization unit 604 stabilizes the image by using the camera vector calculation. However, by using the camera vector, the camera posture is stabilized and the position and orientation stabilization control is performed. It can also be done.
As described above, an artificial intelligence device is attached to an object, and exchanges data such as movement and attributes of the object with other artificial intelligence devices to constitute a composite artificial intelligence device. Since an object to which an intelligent device is attached normally moves, the movement must be accurately grasped. In addition, appropriate control must be performed in order to move as scheduled.
The following description assumes that camera position and orientation control and object position and orientation control are synonymous.

From the three-dimensional position signal of the camera and the three-axis rotation signal (X, Y, Z, [Phi] x, [Phi] y, [Phi] z) obtained by the above-described camera vector detection, the camera position (X, Y, Z,) and the camera posture The shake component (Φx, Φy, Φz) due to the rotation of the camera is extracted, and a correction signal for correcting the camera position can be generated from the difference between the scheduled camera position and the current camera position.
Further, a rotation correction signal for correcting the rotation of the camera can be generated from the difference between the normal camera posture and the current camera posture.
Then, with this position and rotation correction signal, the posture of the camera, the holding mechanism to which the camera is fixed, or the moving object on which the camera is loaded can be controlled and corrected.

In this way, the mechanism that holds the camera itself is controlled by the correction signal obtained in real time from the camera vector calculation, and the camera itself that acquires the image is controlled in position and rotation in real time to stabilize the camera posture. The position and orientation stabilization device can be configured.
In addition, artificial intelligence devices attached to aircraft, vehicles, etc. by using such correction signals can be used not only for rotational attitude control but also for vehicle position control by controlling vehicles with fixed cameras, etc. it can.

Here, the correction signal may be attitude control or position control depending on which combination of X, Y, Z, Φx, Φy, and Φz is used to generate the control signal.
That is, the rotation angle can be controlled to be constant by the rotation correction signal generated from the rotation angles Φx, Φy, Φz of the camera. In general, it is not necessary to fix all of these, and there are cases where it is better to control the rotation while leaving free rotation of one axis.
Further, when the helicopter is hovering in the sky, it can be realized by controlling all of X, Y, Z, Φx, Φy, and Φz.
In addition, position control can be guided to a scheduled route by detecting a deviation from a planned three-dimensional position and feeding it back as a position control signal for a vehicle or the like. By comparing the planned route of the vehicle or aircraft with X, Y, Z, it can be used as position control.

In this way, the camera vector required in the present invention can be used as a position and orientation control device for a moving body such as a vehicle or an aircraft.
In addition, since the real-time property is required for the control of the moving body, as described above, by using a plurality of videos, the video output acquired in real time is directly used for camera vector calculation, and past data and In comparison, the camera vector is calculated in real time (see FIGS. 27 and 28).
Since the high-precision camera vector obtained by the real-time calculation includes the three-dimensional position and the three-axis rotation signal of the camera, it is possible to realize attitude control and position control of the vehicle loaded with the camera.
Further, by combining with a high-accuracy three-dimensional calculation device or the like as shown in FIG. 26, more accurate position and posture control can be performed.

  The composite artificial intelligence apparatus of the present invention has been described with reference to the preferred embodiment. However, the composite artificial intelligence apparatus according to the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the present invention. Needless to say, it is possible to implement this change.

  The present invention can be suitably used in the field of an artificial intelligence apparatus that automatically controls an arbitrary object such as a camera, a vehicle, or a robot based on a moving image captured by a video camera or the like.

It is a block diagram which shows the basic functional structure of each artificial intelligence apparatus which comprises the composite artificial intelligence apparatus of this invention. It is explanatory drawing which shows the relationship between the object and target object of the real world to which the artificial intelligence apparatus shown in FIG. 1 is attached. It is explanatory drawing which shows more concretely the relationship between the object and target object to which the artificial intelligence apparatus shown in FIG. 2 is attached. It is a functional block diagram for demonstrating the more concrete function implement | achieved in each artificial intelligence apparatus shown in FIG. It is a block diagram which shows schematic structure of one Embodiment of each artificial intelligence apparatus which comprises the composite artificial intelligence apparatus of this invention. It is the block diagram which showed the detail of schematic structure of the artificial intelligence apparatus shown in FIG. It is explanatory drawing which shows the specific detection method of the camera vector in the camera vector detection part of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the specific detection method of the camera vector in the camera vector detection part of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the specific detection method of the camera vector in the camera vector detection part of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the designation | designated aspect of the desirable feature point in the detection method of the camera vector by the camera vector detection part of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the camera vector detection part of the artificial intelligence apparatus which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the camera vector detection part of the artificial intelligence apparatus which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the camera vector detection part of the artificial intelligence apparatus which concerns on one Embodiment of this invention, and a camera vector. In the camera vector detection part of the artificial intelligence apparatus which concerns on one Embodiment of this invention, it is explanatory drawing which shows the case where a some feature point is set according to the distance of a feature point from a camera, and a some calculation is repeated. It is explanatory drawing which shows the production | generation process of the unit polygon in the feature point connection polygon production | generation part of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the production | generation process of the unit polygon in the feature point connection polygon production | generation part of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the process which adds a feature point to the part which calculates | requires a detail, and makes it a polygon in the feature point connection polygon production | generation part of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows the comparison of the polygon surface and component surface in the comparison recognition apparatus of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is a principal part block diagram which shows the modified embodiment of the artificial intelligence apparatus shown in FIG. It is a block diagram which shows schematic structure of the meaning understanding judgment apparatus and control apparatus of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the audio | voice guide output from the display apparatus of the artificial intelligence apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the reference data of the reference | standard image | video used as reinforcement of the position and orientation of an object in the composite artificial intelligence apparatus of this invention. It is a block diagram which shows schematic structure of the calculating means of the position / orientation using the reference | standard image | video shown in FIG. It is explanatory drawing which shows the process which calculates | requires the position and orientation of an object using a reference | standard image | video in the composite artificial intelligence apparatus of this invention. It is explanatory drawing which shows the process which calculates | requires the position and orientation of an object using a reference | standard image | video in the composite artificial intelligence apparatus of this invention. It is a block diagram which shows schematic structure of embodiment provided with the highly accurate three-dimensional arithmetic unit in the artificial intelligence apparatus shown in FIG. It is a block diagram which shows schematic structure of one Embodiment provided with several image acquisition apparatus in each artificial intelligence apparatus which comprises the composite artificial intelligence apparatus of this invention. It is the block diagram which showed the detail of schematic structure of the artificial intelligence apparatus shown in FIG. It is explanatory drawing which shows the relationship between the wide-angle visual field camera and narrow-angle visual field camera with which one Embodiment of the composite artificial intelligence apparatus of this invention is equipped. In an artificial intelligence apparatus provided with a plurality of cameras with a wide-angle field of view and a narrow-angle field of view, it is an explanatory diagram when a plurality of narrow-angle field-of-view cameras are installed with overlapping fields of view along the moving direction of the camera. In an artificial intelligence apparatus provided with a plurality of cameras with a wide-angle field of view and a narrow-angle field of view, it is an explanatory diagram in a case where an image is acquired by overlapping a field of view with a plurality of narrow-angle field-of-view cameras while capturing the entire field of view with a wide-angle field-of-view camera. It is explanatory drawing at the time of installing multiple wide-angle visual field cameras which capture the whole visual field in the artificial intelligence apparatus provided with the several camera of a wide-angle visual field and a narrow-angle visual field. It is a block diagram which shows schematic structure of the artificial intelligence apparatus which performs the mobile body detection, stationary coordinate system separation, and mobile body coordinate system separation which concern on one Embodiment of this invention. It is a block diagram which shows schematic structure of the artificial intelligence apparatus which performs the mobile body detection, stationary coordinate system separation, and mobile body coordinate system separation which concern on one Embodiment of this invention. It is explanatory drawing which shows the process of the mobile body detection which performs in the artificial intelligence apparatus shown in FIG.33 and FIG.34, stationary coordinate system separation, and mobile body coordinate system separation. It is explanatory drawing which shows the process of the mobile body detection which performs in the artificial intelligence apparatus shown in FIG.33 and FIG.34, stationary coordinate system separation, and mobile body coordinate system separation. It is explanatory drawing which shows the process of the mobile body detection which performs in the artificial intelligence apparatus shown in FIG.33 and FIG.34, stationary coordinate system separation, and mobile body coordinate system separation.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Artificial intelligence apparatus 101 Self-object attribute storage function 102 Self-object spatial position and orientation recognition function 103 Self-object surrounding object recognition function 104 Self-object situation judgment function 105 Processing process recording function 106 Communication function 107 Continuation processing function 501 Image acquisition device 502 Image feature point calculation device 503 Comparison reference signal selection device 504 Shape component generation device 505 Shape component database device 506 Comparison candidate signal selection device 507 3D map device 508 by component configuration 508 Camera approximate position determination device by GPS 509 Comparison recognition device 510 Coordinate Integration Device 511 3D Map DB Device with Updated Component Configuration 512 Semantic Understanding Judgment Device 513 Control Device 514 Display Device (Output Device)
515 High precision 3D arithmetic unit

Claims (14)

With multiple artificial intelligence devices attached to any object,
Each artificial intelligence device
A three-dimensional space configuration unit configured to integrate the image information of the surrounding space of the object to which the artificial intelligence device is attached obtained by the image acquisition device into a unified coordinate system, and to configure the outline of the three-dimensional shape of the space;
A camera vector indicating a three-dimensional coordinate position and a three-axis rotation angle posture of the camera of the image acquisition device is obtained by a predetermined camera vector calculation , and exists in the three-dimensional space of the space obtained by the three-dimensional space configuration unit, Three-dimensional coordinate data indicating the position, posture and movement of the object to which each artificial intelligence device is attached is acquired with reference to the camera vector , and the acquired three-dimensional coordinate data of the object with reference to the unified coordinate system Is placed in the unified coordinate system "
Unified coordinate system three-dimensional coordinate data of the object has been placed, the three-dimensional coordinate data indicating the moving three-dimensional position and orientation of the object artificial intelligence device which is an image acquired by the image acquisition device is not attached An object position movement additional placement unit that obtains the camera vector as a reference and additionally places the obtained three-dimensional coordinate data of the target in the unified coordinate system;
A database unit that stores in advance a plurality of three-dimensional component data given a three-dimensional shape and predetermined attribute information corresponding to the object and the object;
The three-dimensional part data corresponding to the object and the target object, the three-dimensional shape or camera vector of the three- dimensional part data stored in the database unit, and the viewpoint position of the camera of the image acquisition device of the three-dimensional space configuration unit An attitude narrowing unit that extracts from the three-dimensional coordinate data shown and narrows down the candidate three-dimensional part data based on the approximate arrangement and attitude of the three-dimensional part data ;
A comparison unit that compares the three-dimensional shape data of the object and the object with the narrowed parts;
An object target part specifying unit for specifying a corresponding part based on the comparison mismatch;
At the same time as or after the specification of the part, the three-dimensional position of the part is determined, and the object object part coordinate fixing unit to be rearranged in the unified coordinate system;
Predetermined attribute information including the name of the object pre-stored for the object and predetermined attribute information including the name of the object of the part pre-stored for the part after or simultaneously with or prior to specifying and fixing the part And a component narrowing-down part by an attribute having a component with matching attribute information within a predetermined condition as a component candidate,
An object component attributes comparator additional updating unit for the adds the attribute information of the identified with the parts having undergone or updated in response to a predetermined input operation,
In the unified coordinate system, a part reconstructing three-dimensional space generating unit that reconstructs the three-dimensional space with a plurality of parts and generates a part reconstructing three-dimensional space;
Setting the attribute information to the parts to be component disposed the component reconstructed three-dimensional space in response to a predetermined input operation, a virtual reality by multiple components having a predetermined attribute information corresponding to the real world A virtual space part corresponding to reality for generating a virtual space;
In a three-dimensional coordinates of the real corresponding virtual space, to specify the three-dimensional data indicating the moving three-dimensional position and orientation of the artificial intelligence system shown as three-dimensional coordinates of the camera viewpoint position of the image acquisition device, the A self-recognition unit in a virtual space that acquires the three-dimensional position information of the part centered on the specified position and the attribute information of the part;
A semantic rule providing unit that stores in advance a rule for generating a predetermined meaning from the mutual relationship between the three-dimensional arrangement of a plurality of parts and their attribute information;
A meaning generating unit that generates a predetermined meaning based on the rule from the positional relationship between the artificial intelligence device and the component in the reality-enabled virtual space and the relationship between the attribute information,
A self-situation grasping unit that outputs the meaning generated by the meaning generating unit as information indicating a situation in a real-world three-dimensional space of an object to which the artificial intelligence device in the reality-compatible virtual space is attached;
A time history simulation unit for obtaining information output from the self-status grasping unit a plurality of times along the time history;
An action determination unit that outputs a control signal for controlling an object to which the artificial intelligence device is attached based on information obtained a plurality of times by the time history simulation unit;
Multiple artificial intelligence devices are connected so that data can be sent and received,
Among the plurality of artificial intelligence devices, the attribute information of the object to which each artificial intelligence device is attached, the image information of the surrounding space of the object to which each artificial intelligence device is attached, the three-dimensional spatial information based on the image information, Three-dimensional information indicating the position, posture and movement of the object in which each artificial intelligence device is attached in the three-dimensional space, and the object around which the artificial intelligence device is not attached 3D information indicating 3D position, posture and movement, 3D placement and attribute relationship information of parts centered on the self position of the object to which each artificial intelligence device is attached, and partial information of the semantic information based on it Send and receive those combination information,
In each artificial intelligence device, each information is received from another artificial intelligence device, and the unified coordinate system of the object to which the artificial intelligence device is attached is connected to the three-dimensional position of the other object on the basis of the unified coordinate system. A composite artificial intelligence device characterized in that three-dimensional coordinate data indicating posture and movement is additionally arranged to perform the above-described processes. Any artificial intelligence device among the plurality of artificial intelligence devices is
In each part of the artificial intelligence device by having only a part of each part of the artificial intelligence device, transmitting information to other artificial intelligence devices, or receiving information from other artificial intelligence devices The composite artificial intelligence apparatus according to claim 1, wherein part or all of the processing is performed. A process storage unit that stores all or part of the processes leading to the action determination of the action determination unit;
A type classification unit that classifies and stores processes in the process storage unit into similar processes, and
The composite artificial intelligence device according to claim 1, further comprising a learning device that simplifies the process by reproducing a similar process that has already been obtained. Each of the artificial intelligence devices is
An image acquisition device for acquiring moving images with a video camera;
Recording or primary recording of the video image,
Automatically extract a sufficient number of feature points with image features in the recorded video, automatically track the feature points in the video image,
From the plurality of feature points, the three-dimensional coordinates of each feature point are determined by calculation,
From overlapping relationship of the three-dimensional coordinates of a plurality of the feature points obtained by the calculation to obtain the said predetermined camera vector by statistical processing operation,
In the video, additional feature points in the vicinity of an arbitrary part are extracted and tracked in a moving image,
From the already obtained camera vector, the three-dimensional coordinates and movement vector of the added additional feature point are calculated and output, and if there is a moving body, a moving body vector signal is output,
The unit points are generated by combining the feature points obtained at the same time as the camera vector and the 3D coordinates of the additional feature points obtained by the calculation with the later camera vector as known without contradiction. By combining, a polygon surface group is generated, a plurality of polygon surface groups are generated, a plurality of polygon surface groups obtained from a plurality of frame images are overlapped, and a polygon surface group is added to the gap, Overlays to fill the gaps to generate a combined polygon group, statistical processing generates an average polygon surface, and by repeating this multiple polygon calculation, the target object and the surrounding average polygon surface are continuously extracted, and the target Form the object and its surrounding polygonal 3D shape,
The camera vector signal and the moving body moving vector signal if there is a moving body and the polygon surface shape are generated and output as a feature point surface distribution signal. If there is a moving body, the polygon surface of the moving body An image feature point computing device that generates a shape and outputs a moving body feature point plane distribution signal;
A comparison reference signal selection device that outputs one or both of the feature point plane three-dimensional distribution signal and the video signal output from the image feature point calculation device, as a comparison reference signal;
A shape component generating apparatus that generates a video component obtained by cutting out an object to be predicted in the moving image or a part of a characteristic portion thereof, and a 3DCG component that is an object to be predicted in the moving image or a part thereof. ,
A shape component database device in which a plurality of components generated by the shape component generation device and a part of the surface shape of the object are recorded as 3DCG components;
The camera vector signal obtained by the image feature point calculation device is used to specify the approximate position of the camera to narrow down the location and the like, and the parts that are expected to be present in the vicinity of the approximate position are selected from the shape parts database device. A comparison candidate signal selecting device that selects and outputs as a comparison candidate signal;
The comparison reference signal from the comparison reference signal selection device is compared with the comparison candidate signal from the comparison candidate signal selection device. If they do not match, the comparison candidate signal is selected and compared until they match, A comparison recognition device that determines that the object has been recognized in a one-to-one correspondence between the comparison reference signal and the comparison candidate signal, and identifies and outputs the coordinates and components of the recognized comparison candidate signal;
The respective processes are continuously performed for the areas in each frame of the moving image, and the respective processes are continuously repeated for all the frames of the moving image. And a coordinate integration device that integrates the coordinates of a plurality of recognized parts into a stationary object,
An output device for reconstructing a part or all of the moving image with known parts, and outputting the three-dimensional coordinate position and three-axis rotation posture of the camera obtained as the camera vector therein as data,
A composite artificial intelligence apparatus according to any one of claims 1 to 3. From the three-dimensional reconstruction information of the parts integrated by the coordinate integration device, the position and attribute of the three-dimensional space where the camera is placed are understood, and the object is obtained from information given in advance as the attribute of the object part. Understand the arrangement of a plurality of recognition objects with a single predetermined information and each attribute, or understand the position and attributes of the three-dimensional space where the camera is located from their combined information and attach the artificial intelligence device A semantic understanding determination device for determining the next action of the given object;
The composite artificial intelligence device according to claim 4, further comprising: a control device that controls a camera or a control target device based on an output of the meaning understanding determination device.   The composite artificial intelligence device according to claim 4 or 5, wherein the shape component generation device gives predetermined attribute information to each component when generating the component. A 3D map device having a component configuration in which components of the shape component database device are arranged so as to correspond to the video image;
A rough position is received from predetermined GPS data provided in the comparison candidate signal selection device, and the rough position is set as a temporary position of a 3D map device based on a component configuration, and a plurality of parts near the temporary position are selected. A 3D map array signal for sending a signal to the comparative recognition device;
The composite 3D map apparatus according to any one of claims 4 to 6, further comprising: an updated part configuration that generates an updated 3D map by using an updated part that is identified by the comparison recognition device and whose coordinates are determined. Artificial intelligence device. A reference image obtained by capturing a moving path or a movement range of the object in advance and camera vector calculation data corresponding to each frame image of the reference image are stored,
All or part of the reference image is converted into three-dimensional data by the camera vector calculation data,
The composite artificial intelligence apparatus according to any one of claims 4 to 7, wherein the camera vector is obtained by obtaining a correspondence between the reference image and the moving image. Instead of the reference video, a plurality of feature parts for obtaining correspondence with the moving picture in the reference video is stored in advance as a reference feature part video,
Converting the reference feature part image from the camera vector calculation data to two-dimensional data or three-dimensional data;
The composite artificial intelligence apparatus according to any one of claims 4 to 7, wherein the camera vector is obtained by obtaining a correspondence between the reference feature part image and the moving image. A three-dimensional shape tracking unit for tracking a three-dimensional shape using a part of the three-dimensional shape of data having three-dimensional information generated after the camera vector obtained by the image feature point computing device is a rough camera vector When,
A high-precision camera vector calculation unit that outputs a predetermined high-precision camera vector from the tracking data obtained by the three-dimensional shape tracking unit;
The composite artificial intelligence apparatus according to claim 4, further comprising a high-precision three-dimensional shape calculation apparatus having The image acquisition device has a plurality of synchronized video cameras for acquiring a plurality of video images,
The image feature point calculation device is provided in the same number as the plurality of video cameras,
The composite artificial intelligence apparatus according to any one of claims 4 to 10, wherein the outputs of a plurality of cameras are calculated in an image feature point calculation apparatus corresponding to each camera while being correlated between the same function units. Each of the artificial intelligence devices is
A moving image acquisition unit that acquires a moving image from a camera attached to the object;
A feature point detection / tracking unit that extracts a plurality of feature points of a stationary object in the moving image and tracks the plurality of extracted feature points over a plurality of frame images;
A camera calculation unit for calculating a camera vector indicating the position and orientation of the camera from the tracking result of the feature points;
An image stabilization unit that stabilizes the image by stopping shaking caused by rotation of the image from the camera vector;
A pair of traces of the feature points are rearranged on the image stabilized by the image stabilization unit, and extended lines of the traces of the feature points on the stabilized image intersect in a stationary coordinate system. A stationary extended intersection extraction unit for obtaining an extended intersection of
The stabilized image is divided into small-area divided images, corresponding points are obtained over a plurality of frame images for each divided image, and stationary coordinates are determined depending on whether each divided image passes through the extended intersection. A divided image classifying unit that classifies the system or other types,
A stationary coordinate system classification extraction unit that extracts and temporarily records a divided image determined as a stationary coordinate system by the divided image classification unit based on the extended intersection;
A non-stationary coordinate system classification extraction unit that extracts and temporarily records divided images determined by the image classification unit depending on whether or not they belong to the extended intersection; and
For the divided image determined to be the stationary coordinate system obtained by the stationary coordinate system classification extraction unit, the corresponding point is obtained on the extension connecting the extended intersection point and the center of the divided image, thereby obtaining the three-dimensional coordinates of the divided image. A coordinate system 3D part,
A moving object divided image tracking unit for tracking the moving object divided image separated by the classification extracting unit other than the stationary coordinate system over a plurality of frames;
From the tracking trajectory by the moving object divided image tracking unit, a tracking extended intersection of the moving object divided image is obtained, and for each obtained moving object extended intersection, the moving object divided image is classified by moving object, ,
The moving object for obtaining the shape of the moving object and its coordinates by obtaining the three-dimensional coordinates of the divided image for each moving object from the divided images for each moving object classified by the moving object extension intersection classification unit and the tracking result thereof. Another three-dimensional part,
A common known length obtained by the three-dimensional moving body, or a scale matching section that shares the scale of each moving body with a stationary coordinate system based on an object common to a plurality of cameras,
A coordinate integration unit that integrates the three-dimensional coordinates of each moving body and the coordinates of the stationary coordinate system that have been scaled in the scale adjustment unit;
The composite artificial intelligence device according to claim 1, comprising: From the divided image tracking locus for each moving object obtained by the extended intersection classification unit for each moving object, a camera vector for each virtual moving object is detected and used as a virtual camera vector for each moving object, and the plurality of virtual cameras for each moving object A coordinate system determining unit for each moving body that performs coordinate conversion so that the vector coordinate system matches a camera vector of a known stationary coordinate system;
13. The composite artificial intelligence apparatus according to claim 12, wherein the coordinate system for each moving object is transformed into a stationary coordinate system, and the moving object is integrated into the stationary coordinate system for each moving object. From a moving object-specific extended intersection obtained by the moving object-specific extended intersection classification unit or a camera vector, a moving object-specific coordinate determination unit obtains a coordinate system of an arbitrary moving object,
The camera direction is controlled with reference to an arbitrary position and orientation determined in relation to the coordinate system for each moving body so as to fix the extended intersection of the arbitrary moving body, or an image without controlling the camera. The composite artificial intelligence apparatus according to claim 12 or 13, further comprising: a lock-on unit for each moving body that displays a coordinate in a coordinate system of the arbitrary moving body.