WO2021125395A1

WO2021125395A1 - Method for determining specific area for optical navigation on basis of artificial neural network, on-board map generation device, and method for determining direction of lander

Info

Publication number: WO2021125395A1
Application number: PCT/KR2019/018118
Authority: WO
Inventors: 이훈희; 정다운; 최한림; 류동영; 주광혁
Original assignee: 한국항공우주연구원; 한국과학기술원
Priority date: 2019-12-18
Filing date: 2019-12-19
Publication date: 2021-06-24
Also published as: KR102314038B1; KR20210078326A

Abstract

Various embodiments provide a method for determining a specific area, a method for generating an on-board map, and a method for determining a direction of a lander. The method for determining a specific area is performed by a computing device, and comprises the steps of: determining an observation region and a plurality of candidate areas within the observation region; labeling location information on each of a plurality of images in which at least a part of the observation region is shown; training an artificial neural network based on a convolutional neural network on the basis of the plurality of images labeled with the location information, so that each of the plurality of candidate areas can be searched for in each of the plurality of images; evaluating searchability of each of the plurality of candidate areas on the basis of the trained artificial neural network; and determining at least a part of the plurality of candidate areas as specific areas on the basis of the searchability of each of the plurality of candidate areas.

Description

A method for determining a singular region for optical navigation based on an artificial neural network, an on-board map generator, and a method for determining the direction of a lander

The present invention relates to a method for determining a specific area for optical navigation based on an artificial neural network, an onboard map generation device, and a method for determining the direction of a lander, and more specifically, to a satellite navigation system such as the moon, Mars, and Jupiter. A method and computer program for determining a robust singularity region based on deep learning to use optical navigation on a non-existent planet, an apparatus for generating an onboard map including the determined singularity region, and the onboard map thus generated It relates to a method to determine the direction of a lander using

As the space age approaches, attempts are being made to more precisely probe extraterrestrial planets (hereinafter referred to as 'planets') such as the moon, Mars, and Jupiter. On Earth, it is possible to know its position relatively accurately using a satellite navigation system, but on a planet, there are limited ways to know its position.

In order to move or land to a desired position on a planet, an aircraft such as a spaceship or a lander must know its own position. An optical navigation method for estimating the current position by storing a picture of the planet's surface in the form of a map, and comparing a picture of the planet's surface at the current location with the stored map may be used.

Because planets do not have an atmosphere, unlike Earth, the shape of the surface of the Earth hardly changes, so it is possible to estimate their position based on pictures of the surface of the earth. However, unlike Earth, there are no artificial landmarks on the planet, only the terrain.

The craters formed on the surface of the planet are used in optical navigation as special areas such as landmarks, but when the sun is low as in the poles of the planet, the shadows of the mountainous terrain are long. The crater may not be identified. Also, craters may not exist in some areas. In addition, since the direction of the photo changes depending on the direction the vehicle approaches, it is difficult to compare it with the stored map.

If it is possible to determine a singular region that can be strongly distinguished from other regions even when the position of the sun or the position of a vehicle changes, it is possible to accurately estimate its position on a planet using optical navigation.

It is an object of the present invention to provide a method for determining specific areas or topography on the surface of a planet that can function as a landmark for optical navigation despite changes in the position of the sun and the position of a vehicle.

Another technical object of the present invention is to provide an apparatus for generating an onboard map mounted in a memory to perform optical navigation in an aircraft flying around a planet or a lander intended to land on a landing point of a planet.

Another technical object of the present invention is to provide a method for determining the direction of a lander by using an onboard map and a camera image in order for the lander to land at a preset landing point of a planet.

As a technical means for achieving the above-described technical problems, the method for determining a singular region according to an aspect of the present invention is performed by a computing device, and includes the steps of determining an observation area and a plurality of candidate areas within the observation area, the observation area labeling each of a plurality of images in which at least a portion of the image is displayed, and convolution based on the plurality of images labeled with the position information to search each of the plurality of candidate regions in each of the plurality of images Training an artificial neural network based on a neural network, evaluating the searched performance of each of the plurality of candidate areas based on the trained artificial neural network, and based on the searched performance of each of the plurality of candidate areas and determining at least some of the plurality of candidate regions as singular regions.

According to an example, the plurality of candidate areas may be distributed so as not to overlap each other in the observation area.

According to another example, the plurality of candidate areas may have a preset shape and may be arranged adjacent to each other in the observation area.

According to another example, the plurality of candidate regions may be circular having a first diameter, and may be arranged adjacent to each other in the observation region.

According to another example, the plurality of images of the observation area may include satellite images photographed by at least one artificial satellite flying along an orbit having a period on the observation area.

According to another example, the plurality of images of the observation area may include images photographed by an aircraft flying over the observation area.

According to another example, the plurality of images of the observation area may include images of the observation area from the earth.

According to another example, the plurality of images of the observation area are synthesized images generated based on a modeled topography of the observation area, a position of the sun with respect to the observation area, and a pose of a camera photographing the observation area may include

According to another example, the method for determining the singular area may further include determining a landing point at which the lander will land and a landing trajectory for landing at the landing point. The observation area may be determined according to the landing trajectory.

According to another example, the method for determining the singular region may further include enlarging or reducing the plurality of images based on an altitude of the landing trajectory on the observation area and a spatial resolution of a camera mounted on the lander. .

According to another example, the location information may include location coordinates of a photographing area displayed in each of the plurality of images. Image coordinates of each of the plurality of candidate regions in each image may be extracted based on the coordinates of the location of the capturing region labeled in the respective images.

According to another example, the location information may include image coordinates of the plurality of candidate regions in each image.

According to another example, the labeling includes: additionally labeling the position of the sun with respect to the observation area at a viewpoint corresponding to each of the plurality of images and a pose of the camera with respect to the observation area on each of the plurality of images may include.

According to another example, the training of the artificial neural network may include inputting a first image from among the images labeled with the location information to the artificial neural network, as an output of the artificial neural network, in the first image. receiving estimated image coordinates of the plurality of candidate regions, and image coordinates of the plurality of candidate regions labeled in the first image and estimated image coordinates of the plurality of candidate regions in the first image It may include training the artificial neural network so that the difference between the two is minimized.

According to another example, the training of the artificial neural network may include receiving, as an output of the artificial neural network, estimated sizes of the plurality of candidate regions in the first image, and sizes of the plurality of candidate regions. The method may further include training the artificial neural network to minimize a difference between the values and the estimated sizes of the plurality of candidate regions in the first image.

According to another example, evaluating the searched performance of each of the plurality of candidate areas may include evaluating the searched performance of a first candidate area among the plurality of candidate areas. Evaluating the search performance of the first candidate region may include inputting each of a plurality of test images in which at least a part of the observation region appears to the trained artificial neural network, as an output of the trained artificial neural network, the receiving estimated image coordinates of the plurality of candidate regions in each of a plurality of test images, and _{calculating an F 1} score based on an output of the trained artificial neural network, and calculating the F ₁ score based on the calculated F 1 score The method may include determining classification performance of the first candidate region.

According to another example, the evaluating the search performance of the first candidate region may include image coordinates of the first candidate region labeled in each test image and estimation of the first candidate region in each test image. and determining the accuracy performance of the first candidate region based on the position difference between the image coordinates.

According to another example, the evaluating the search performance of the first candidate region may include: based on the position of the sun with respect to the observation region labeled in each of the plurality of test images, according to the azimuth and elevation of the sun. The method may further include displaying the classification performance and the accuracy performance of the first candidate region as a classification performance graph and an accuracy performance graph, respectively.

According to another example, the determining of at least some of the plurality of candidate regions as singular regions may include determining the singular regions based on a classification performance graph and an accuracy performance graph for each of the plurality of candidate regions. may include.

According to another example, the method for determining the singular area may further include determining a landing point at which the lander will land and a landing trajectory for landing at the landing point. The determining of at least some of the plurality of candidate regions as singular regions may include: based on a classification performance graph and an accuracy performance graph for each of the plurality of candidate regions, and a first landing date and time of the lander determining at least some of the candidate regions as first singular regions; generating a first onboard map including the first singular regions within the observation region; and a classification performance graph for each of the plurality of candidate regions. determining at least some of the plurality of candidate regions as second singular regions based on an accuracy performance graph, and a second landing date and time of the lander, and a first comprising the second singular regions within the observation region 2 generating an onboard map.

As a technical means for achieving the above-described technical problems, the computer program according to an aspect of the present invention is stored in a medium to execute the above-described specific region determination methods using a computing device.

As a technical means for achieving the above technical problems, an onboard map generating apparatus according to an aspect of the present invention includes a memory for storing a plurality of images showing at least a part of an observation area, and an artificial neural network based on a convolutional neural network, and at least It contains one processor. The at least one processor determines the observation area and a plurality of candidate areas within the observation area, labels location information on each of the plurality of images, and selects each of the plurality of candidate areas from each of the plurality of images. In order to search, the artificial neural network is trained based on the plurality of images labeled with the location information, and searched performance of each of the plurality of candidate regions is evaluated based on the trained artificial neural network, and the plurality of candidates and determine at least some of the plurality of candidate regions as singular regions based on the searched performance of each of the regions, and generate the onboard map including the singular regions within the observation region.

As a technical means for achieving the above-described technical problems, the method for determining the direction of the lander according to an aspect of the present invention is performed by a processor of the lander, and while flying according to the landing trajectory, a preset observation area at a preset altitude generating an observation region image by photographing; searching for singular regions within the observation region of an onboard map stored in a memory in the observation region image; and determining a direction of the probe based on a difference in arrangement directions of the singular regions. The onboard map may include: determining an observation area and a plurality of candidate areas within the observation area; labeling location information on each of a plurality of images in which at least a part of the observation area appears; training a convolutional neural network-based artificial neural network based on the plurality of images labeled with the location information to search for each of the candidate regions of evaluating search performance; determining at least some of the plurality of candidate areas as the singular regions based on the searched performance of each of the plurality of candidate regions; and identifying the singular regions within the observation region. It is generated by a computing device that performs the step of generating the onboard map including.

According to the present invention, in spite of changes such as the position of the sun and the position of the vehicle, specific areas or topography on the surface of the planet that can function as a landmark for optical navigation can be determined. Accordingly, even in a region with a lot of shade, such as the poles of the moon, specific regions or topography that can be distinguished from other regions or topography can be used as landmarks.

An onboard map that is loaded into memory can be created to perform optical navigation on a vehicle flying around a planet or a lander attempting to land on a planet's landing site. Since the moon has no atmosphere, there is little change in terrain, and the onboard map can be used semi-permanently.

In order for the lander to land at a preset landing point on the planet, the onboard map and camera image can be used to accurately determine the lander's direction.

1 exemplarily shows an orbiter revolving around the moon and a lander landing at a landing point on the moon in accordance with the present invention.

Figure 2 exemplarily shows a lander landing on the lunar landing site in accordance with the present invention.

3 schematically shows the internal configuration of a lander according to the present invention.

4 schematically shows the operation of the processor of the lander according to the invention;

5 schematically shows a computing device for determining singular regions based on images of an observation region of the moon and generating an on-board map including the singular regions according to the present invention.

6A to 6C exemplarily show configurations of candidate areas in an observation area according to various embodiments of the present disclosure.

7 schematically shows an internal configuration of a computing device according to the present invention.

8 is an exemplary block diagram of a control unit according to embodiments of the present invention.

9 is a block diagram of a data learning unit according to an embodiment of the present invention.

10 is a block diagram of a data recognition unit according to an embodiment of the present invention.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present disclosure pertains can easily implement it. However, the technical spirit of the present disclosure may be modified and implemented in various forms, and thus is not limited to the embodiments described herein. In the description of the embodiments disclosed in the present specification, when it is determined that a detailed description of a related known technology may obscure the gist of the present disclosure, a detailed description of the known technology will be omitted. The same or similar components are given the same reference numerals, and overlapping descriptions thereof will be omitted.

In this specification, when an element is described as being "connected" with another element, it includes not only the case of being "directly connected" but also the case of being "indirectly connected" with another element interposed therebetween. When an element "includes" another element, it means that another element may be further included without excluding another element in addition to other elements unless otherwise stated.

Some embodiments may be described in terms of functional block configurations and various processing steps. Some or all of these functional blocks may be implemented in various numbers of hardware and/or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or by circuit configurations for a given function. The functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks of the present disclosure may be implemented as an algorithm running on one or more processors. A function performed by a functional block of the present disclosure may be performed by a plurality of functional blocks, or functions performed by a plurality of functional blocks in the present disclosure may be performed by one functional block. Also, the present disclosure may employ prior art for electronic configuration, signal processing, and/or data processing, and the like.

An aircraft flying using optical terrain-referenced absolute navigation needs proper landmark information along the flight path. When the vehicle flies close to a planet such as the moon, the crater can be selected as an intuitive landmark for optical navigation. However, in the case of the highland region of the moon, it is difficult to reliably detect the craters due to the large shaded area, and the craters may not function as landmarks for optical navigation.

The present invention proposes a method for determining singular areas that can be used as landmarks of optical navigation even in rough terrain. The present invention proposes a method for determining a flight plan using singular regions determined as landmarks of optical navigation. The present invention creates an onboard map including specific regions and proposes a method of flying around the moon or landing on the moon using this.

In order to determine a good landmark, a convolutional neural network (CNN)-based object detector is trained to distinguish similar landmark candidates from each other even under various lighting environments. A convolutional neural network is used to predict the detectability of landmarks along flight paths on any day within a year. A date with a high probability of detection may be determined, and a mission plan may be determined with reference to this date.

The present invention determines singular regions that can function as landmarks for optical navigation on the surface of planets such as the moon, Mars, and Jupiter, where navigation satellites do not exist, and an onboard map mounted on an airship or lander based on the determined singular regions is to create

In the detailed description of the present specification, extraterrestrial planets such as the moon, Mars, and Jupiter are collectively referred to as a 'moon' for easy understanding of the present invention. However, the present invention is not used only for exploration around the moon or landing on the moon, and may be applied to other planets such as Mars or Jupiter within the equivalent scope of the present invention.

In addition, although the detailed description of the present specification is illustratively focused on a probe for landing on the moon, the present invention is equally applicable to an orbiter or airship flying in lunar orbit.

In the present specification, training an artificial neural network based on a convolutional neural network (CNN) has substantially the same meaning as learning an artificial neural network using training data. Also, in the present specification, a pose is a concept including a position and a direction. For example, the pose of the lander includes the position of the lander and the orientation of the lander.

Referring to FIG. 1 , the orbiter SC flies along the orbit OR around the moon M. The orbiter SC may load the on-board map generated according to the present invention in a memory, and the processor of the orbiter SC performs the flight direction and/or attitude ( attitude) can be determined. If the determined flight direction and/or attitude differs from the planned flight direction and/or attitude, it may self-correct.

The lander LL may land at a landing point LS of the moon M along a predetermined landing trajectory LT. The lander LL may photograph predetermined observation areas OA1 , OA2 , and OA3 while descending to the landing point LS. The lander LL may load the on-board map generated according to the present invention in a memory, and the processor of the lander LL may store surface images of the observation areas OA1, OA2, OA3 and the observation areas OA1, OA2, The flight direction and/or attitude may be determined based on the onboard map of OA3).

If the lander LL is not landing along the landing trajectory LT, the lander LL can self-correct its landing direction or attitude so that it can land at the landing point LS.

The onboard map stores information about specific objects (or regions) that can serve as landmarks used for optical topographic reference navigation. Singular objects (or regions) are determined for each of the observation regions OA1 , OA2 , OA3 . The specific objects (or areas) may vary depending on the date of landing. According to another example, first and second onboard maps each including first and second unique objects (or regions) different from each other according to the expected landing date are generated, respectively, and the lander ( LL) can be stored in the memory. The first onboard map may be circumferentially prepared, and the second onboard map may be preliminary.

In the present specification, the singular object (or region) refers to an object (or region) that can be distinguished from other objects (or regions) even in variously changing environments in images captured by the observation regions OA1, OA2, and OA3. A singular object actually corresponds to a specific area with a known location within the observation area (OA1, OA2, OA3), and can be understood as a set of specific pixels corresponding to this specific area in the image taken of the observation area (OA1, OA2, OA3). can In this specification, a singular object is referred to as a singular region within the observation areas OA1, OA2, and OA3. A singular area can be distinguished from other areas by the topography of that area.

In the image taken from the observation area (OA1, OA2, OA3), a singular object is another object under various lighting conditions determined by the position (altitude and azimuth) of the sun in the corresponding observation area (OA1, OA2, OA3) of the moon. can be distinguished from

The processor mounted on the lander (LL) searches for singular objects within the observation areas (OA1, OA2, OA3) recorded on the onboard map through correlation analysis or RANSAC algorithm in images captured in the observation areas (OA1, OA2, OA3). By doing so, the attitude of the lander LL can be estimated.

In this example, the observation areas OA1 , OA2 , and OA3 are illustrated as being three, but this is only exemplary, and may be four or more, and may be two or less. When referring to any one of the observation areas OA1, OA2, and OA3, it is denoted as the observation area OA.

Referring to FIG. 2 , the lander LL may descend to the landing point LS at an altitude of, for example, 15.24 km along the landing trajectory LT. The landing point LS may be adjacent to the south pole. For example, it may be 89.98 degrees south latitude and 0.02 degrees east longitude. To this end, the landing could begin at 74.958 degrees south latitude and 0.02 degrees east longitude. The lander LL can reach the landing point LS while flying in the exact south direction.

Pre-set observation areas OA1 and OA3 that can be photographed according to the landing trajectory LT may be photographed. For example, the first observation area OA1 is an area that passes when flying at an altitude of about 10 km according to the landing trajectory LT, and the third observation area OA3 is an area that passes when flying at an altitude of about 3 km according to the landing trajectory LT. It could be a passing area.

In the above example, the observation areas OA1 and OA3 may be located between 74.958 degrees south latitude and 89.98 degrees south latitude, and may be located at 0.02 degrees east longitude. For example, the first observation area OA1 may be located at, for example, 85.39 degrees south latitude.

The polar region of the moon (M) has a high possibility of water presence because the altitude of the sun (S) is low, and the pole region of the moon (M) has mountain peaks that can always communicate with the earth, so it is worth exploring compared to other regions is known to be high.

Because the latitudes of the observation areas OA1 and OA3 are high, the altitude θ of the sun S may be 2 degrees or less, for example, between 0.330 degrees and 1.898 degrees. The azimuth φ of the sun S may be between 0 degrees and 360 degrees.

Referring to FIG. 3 , the lander 10 includes a processor 11 , a memory 12 , a camera 13 , a sensor 14 , and a thruster 15 . Lander 10 may further include components such as star trackers, devices according to a planned mission, and the like. The lander 10 may include only some of the above-described components depending on the design.

The lander 10 may be, for example, a lunar lander LL shown in FIG. 2 , and may fly along a landing trajectory LT to land at a landing point LS of the moon M.

The processor 11 is responsible for overall control of the lander 10 , and may control the lander 10 so that the lander 10 can fly along the landing trajectory LT and land at the landing point LS.

The memory 12 stores instructions for operating the processor 11 . In addition, information about the landing trajectory LT is stored in the memory 12 . The memory 12 includes singular areas within the observation areas OA to determine whether the lander 10 is flying in the planned direction to land at the landing point LS or is maintaining the planned attitude at the landing trajectory LT. An onboard map containing may be stored.

The lander 10 may start to land in order to land at the landing point LS while orbiting around the moon M. Although the landing plan determines the set landing date, other landing dates may be prepared in advance in case of failure to land on that landing date. The landing date on which the preferential landing attempt is made is referred to as the first landing date, and the preliminary landing date prepared in advance in case of failure to land on the first landing date is referred to as the second landing date. In addition to the second landing date, third and fourth landing dates may be predetermined. In this specification, the landing date does not mean only the corresponding date, but includes the landing date and time including the landing time of the corresponding date.

The memory 12 stores a first onboard map of observation areas OA including first singular areas selected according to the solar altitude of the first landing date, and a first onboard map selected according to the solar altitude of the second landing date. A first onboard map of the observation regions O including the two singular regions may be stored. The processor 11 may use the first onboard map on the first landing date and use the second onboard map on the second landing date. The observation area OA of the first onboard map and the observation area OA of the second onboard map may not be identical to each other. In addition, the first singular regions of the first onboard map and the second singular regions of the second onboard map may be the same, some may be the same, or all may be different.

The camera 13 may be mounted on the lander 10 to photograph the surface of the moon M. The camera 13 is mounted at a preset position on the lander 10 and may be oriented in a preset direction. For example, the camera 13 may be set to point in the direction of the center of the moon M, or may point in a fixed direction with respect to the lander 10 .

The camera 13 may be a two-dimensional camera and may generate a black-and-white image. According to another example, the camera 13 may be a one-dimensional line camera. The camera 13 may be a color camera that generates a color image. The angle of view of the camera 13 may be, for example, 80 degrees, and the aspect ratio of the image generated by the camera 13 may be, for example, 1. The resolution of the camera 13 may be, for example, 416 x 416.

The camera 13 may photograph the earth's surface of the moon M under the control of the processor 11 , and may provide an image of the earth's surface to the processor 11 .

The sensor 14 may detect the state of the lander 10 . For example, the sensor 14 may include at least one of an inertial sensor, an acceleration sensor, a gravity sensor, an altitude sensor, and a temperature sensor. The sensor 14 may detect the altitude of the lander 10 . The sensor 14 may provide the sensed value to the processor 11 .

The thruster 15 may provide a force for changing the attitude or flight direction of the lander 10 under the control of the processor 11 .

Referring to FIG. 4 together with FIG. 3 , the processor 11 operates the camera 13 when it is determined that the lander 10 is located on the observation area OA according to the landing procedure instructions stored in the memory 12 . It is possible to obtain an image 13p of the surface of the moon M by doing this. The processor 11 may determine that the lander 10 is located on the observation area OA by sensing the altitude of the lander 10 through the sensor 14 .

The processor 11 loads the onboard map 12m of the corresponding observation area OA stored in the memory 12, the onboard map 12m of the observation area OA and the ground surface image 13p of the observation area OA) can be compared. The processor 11 may search for singular areas SRO in the observation area OA recorded on the onboard map 12m from the surface image 13p of the observation area OA through correlation analysis or RANSAC algorithm. . By comparing the arrangement of the singular regions SRO found in the ground surface image 13p with the arrangement of the singular regions SRO on the onboard map 12m, the processor 11 determines the current flight direction or attitude of the lander 10 . can be decided

According to another example, the processor 11 may include a pre-trained convolutional neural network-based object detection model, and by inputting a ground image 13p and an onboard map 12m to the object detection model, the lander 10 may determine the current flight direction or attitude of

The processor 11 compares the sensed current flight direction or attitude with a direction or attitude according to a preset landing plan, and if they are different, using the thruster 15 to correct the attitude or flight direction of the lander 10 can Accordingly, the lander 10 can land accurately at the predetermined landing point LS according to the predetermined landing plan.

Hereinafter, a method and apparatus for determining singular regions and a method and apparatus for generating an onboard map including singular regions according to the present invention will be described.

5 schematically shows a computing device for determining singular regions based on images of an observation region of the moon and generating an onboard map including the singular regions according to the present invention.

Referring to FIG. 5 , a plurality of images 200 are input to the computing device 100 , and the computing device 100 generates an onboard map 300 based on the plurality of images 200 .

The images 200 are images of the observation area OA of the moon M. Some of the images 200 may include the entire observation area OA. According to another example, some of the images 200 may include only a portion of the observation area OA. That is, only a part of the observation area OA may appear in the image 200 . All of the magnifications of the images 200 may be constant or different from each other.

The images 200 may include training images for training an object detection model based on a convolutional neural network in the computing device 100 , and test images for determining specific regions by testing the object detection model. The images 200 may further include verification images for verifying the validity of the object detection model. In the present specification, an object detection model based on a convolutional neural network may be referred to as an artificial neural network or a lunanet.

As described above, the observation area OA is determined based on a predetermined landing point LS and a landing trajectory LT for landing at the landing point LS. Landing dates for landing at the landing point LS may also be predetermined. The exact position of the observation area OA is known in advance.

The observation area OA may be temporarily determined. When a preset number of singular areas SRO does not exist in the observation area OA, the observation area OA may be changed. According to another example, when a preset number of singular areas SRO does not exist in the observation area OA, the configuration of the candidate areas (CRO of FIG. 6 ) may be changed.

According to an embodiment, the images 200 may include satellite images captured by at least one artificial satellite flying along an orbit having a period on the observation area OA. The images 200 may include images captured by an aircraft flying over the observation area OA. The images 200 may include images obtained by photographing the observation area OA on the Earth.

According to another embodiment, the images 200 are generated based on the modeled topography of the observation area OA, the position of the sun with respect to the observation area OA, and a pose of a camera photographing the observation area OA. Composite images may be included. According to an example, the number of synthesized images generated for the observation area OA may be, for example, hundreds to tens of thousands.

The modeled topography of the observation area OA is captured by satellite images taken by at least one artificial satellite flying along an orbit having a period on the observation area OA, and by a vehicle flying over the observation area OA. The image may be generated based on at least one of captured images and images of the observation area OA on the Earth.

The position of the sun with respect to the observation area OA may be set within a preset range based on the position of the observation area OA. The composite images may be generated while changing the position of the sun with respect to the observation area OA within a preset range.

The pose of the camera capturing the observation area OA may be set within a preset range based on the landing trajectory LT. Synthetic images may be generated while changing a pose of a camera that captures the observation area OA within a preset range. For example, while changing the position of the camera with respect to the observation area OA, the synthesized images may be generated while also changing the direction of the camera facing the observation area OA.

Synthetic images may be generated based on a 3D topography in which the observation area OA is measured. Synthetic images may be generated based on the surface reflectivity and material of the modeled terrain. Synthetic images may be generated based on a modeled atmospheric model. The composite images may be generated based on at least one of a resolution of a camera, a field of view, and a lens.

For example, the synthesized images include the actual three-dimensional topography or modeled topography of the observation area (OA), the modeled surface reflectance and material, the modeled atmospheric model, the position of the sun with respect to the observation area (OA), and the observation area ( OA) may be generated based on at least a part of a pose position and orientation direction, resolution, field of view range, and lens of a camera photographing the OA).

The computing device 100 may determine the observation area OA and a plurality of candidate areas (CRO of FIG. 6 ) within the observation area OA. The computing device 100 generates an object detection model based on the images 200 , and uses the object detection model to create a singular area with particularly high search performance compared to other candidate areas CRO within the observation area OA. (SRO of FIG. 4 ) may be determined. The computing device 100 may generate the onboard map 300 including the singular areas SRO determined as above in the observation area OA. The onboard map 300 is stored in the memory 12 of the lander 10 of FIG. 3, for example, as described above, and can be used to accurately land the lander 10 at the landing point LS of the moon M. .

Below, the internal configuration and operation of the computing device 100 will be described in more detail. Before describing the computing device 100 in detail, candidate areas of the observation area OA will be described first.

6A to 6C exemplarily show configurations of candidate areas in an observation area OA according to various embodiments of the present disclosure.

Referring to FIG. 6A , candidate areas CRO1 , CRO2 , ..., CROi in the observation area OA according to an exemplary embodiment are illustrated. The candidate regions CRO1 , CRO2 , ..., CROi may be referred to as candidate regions CRO.

The candidate areas CRO may be selected by a user within the observation area OA. The candidate areas CRO may be distributed so as not to overlap each other within the observation area OA. Although the candidate regions CRO are illustrated as rectangles in FIG. 6A , these are exemplary and may be circular, triangular, hexagonal, or irregular polygons. At least some of the candidate regions CRO may be inclined. The candidate regions CRO may have different shapes.

Since the user knows the exact location of the observation area OA, the user also knows the exact location of the candidate areas CRO. When the candidate areas CRO are rectangular, at least one of a center position, a horizontal length, a vertical length, and a slope may be determined to specify the candidate area CRO.

Although five candidate areas CRO are illustrated in FIG. 6A , the number of candidate areas CRO may be 4 or less or 6 or more. However, the number of candidate regions CRO may be greater than or equal to a preset number. Here, the preset number may be three or more so that the attitude of the lander can be determined based on the arrangement of the specific regions searched for in the image of the ground surface captured by the lander. As the preset number is set to be larger, the accuracy of the attitude of the lander increases, but all of them may not be detected in the image of the earth's surface.

Referring to FIG. 6B , candidate areas CRO1 , CRO2 , ..., CROi in the observation area OA according to another exemplary embodiment are illustrated. The candidate regions CRO1 , CRO2 , ..., CROi may be referred to as candidate regions CRO.

The candidate areas CRO may all have a preset shape and may be arranged adjacent to each other in the observation area OA. In the example of FIG. 6B , the candidate regions CRO are illustrated as having a regular hexagon, but this is exemplary and may have other shapes such as a square, a triangle, a rhombus, and the like.

The candidate areas CRO may be arranged so that their boundaries contact each other within the observation area OA. However, an empty space may exist between the candidate regions CRO. An area other than the candidate areas CRO among the observation area OA may be referred to as a background area. When the candidate regions CRO are rectangular, the background region may not exist.

Candidate areas CRO may not be set in the edge area of the observation area OA and may be left as a background area. Even if the lander attempts to capture the observation area OA, only a part of the observation area OA may be photographed depending on the attitude of the lander. That is, a part of the edge area of the observation area OA may not be photographed. In consideration of this point, it may be advantageous that the singular areas SRO are not located in the edge area of the observation area OA. To this end, candidate areas CRO may not be set in the edge area of the observation area OA.

The candidate regions CRO may all have the same size and the same shape. In this case, the candidate regions CRO may be specified only by the central position and one length. One length may be set as one of a distance between a center point and a vertex, a length of one side, or a distance between opposing vertices.

Although 39 candidate regions CRO are exemplarily illustrated in FIG. 6B , the number of candidate regions CRO may be less or more. When the candidate regions CRO are set as shown in FIG. 6B , a preset number of singular regions SRO is not found, and at least one of the position, number, shape, and arrangement of the candidate regions CRO is changed. Thereafter, the computing device 100 may again find a preset number of specific regions SRO from among the changed candidate regions CRO.

Referring to FIG. 6C , candidate areas CRO1 , CRO2 , ..., CROi in the observation area OA according to another exemplary embodiment are illustrated. The candidate regions CRO1 , CRO2 , ..., CROi may be referred to as candidate regions CRO.

The candidate areas CRO are all circular having a preset radius, and may be arranged adjacent to each other in the observation area OA. The candidate areas CRO may be arranged so that their boundaries contact each other within the observation area OA. However, the candidate regions CRO may be arranged to be spaced apart from each other. Candidate areas CRO may not be set in the edge area of the observation area OA.

The candidate regions CRO may all have the same radius. In this case, the candidate regions CRO may be specified only by the central position and the radial length.

Although 69 candidate regions CRO are illustrated in FIG. 6C by way of example, the number of candidate regions CRO may be less or more. When the candidate regions CRO are set as shown in FIG. 6C , a preset number of singular regions SRO is not found, and at least one of the position, number, shape, and arrangement of the candidate regions CRO is changed. Thereafter, the computing device 100 may again find a preset number of specific regions SRO from among the changed candidate regions CRO.

For example, the positions of the candidate regions CRO may be horizontally moved by the radial length of the candidate regions CRO. The number of candidate regions CRO may be changed by increasing or decreasing the radial length of the candidate regions CRO. The shape of the candidate regions CRO may be changed to a hexagon, a rectangle, a triangle, etc. as shown in FIG. 6B . A separation distance between the candidate regions CRO may be adjusted.

The candidate areas CRO are set by the user in the observation area OA. Since the user knows the exact location of the observation area OA, the user also knows the exact location of the candidate areas CRO.

Referring to FIG. 7 , the computing device 100 includes a control unit 110 , a memory 120 , and a database (DB, 130 ). The controller 110 , the memory 120 , and the DB 130 may exchange data with each other through a bus.

The computing device 100 may further include additional components such as, for example, a communication module, an input/output device, and a storage device, in addition to the components illustrated in FIG. 7 .

The controller 110 typically controls the overall operation of the computing device 100 . The controller 110 may perform basic arithmetic, logic, and input/output operations, and execute, for example, program code stored in the memory 120 , for example, an object detection model based on a convolutional neural network. The controller 110 may be at least one processor.

Although computing device 100 is shown as one device, computing device 100 may be more than one computing device. The controller 110 may be two or more processors.

The memory 120 is a recording medium readable by the processor 110 and may include a non-volatile mass storage device such as a RAM, a ROM, and a disk drive. The memory 120 may store an operating system and at least one program or application code. In the memory 120 , data for implementing an object detection model based on a convolutional neural network for searching each of the candidate regions CRO in each of the images 100 , a program code for training the object detection model, and an object trained in this way A program code for evaluating the searched performance of each of the candidate regions using the detection model, a program code for determining the singular regions based on the searched performance of each of the candidate regions, and an onboard map including the determined singular regions are generated A program code, etc. for performing the operation may be stored.

The DB 130 is a recording medium readable by the processor 110 , and may include a non-volatile large-capacity recording device such as a disk drive. The DB 130 may store training images used for training the object detection model and test images input to the object detection model to evaluate the search performance of candidate regions. Both the training images and the test images may be included in the images 200 of FIG. 5 .

The images 200 may be stored in the DB 130 as they are without processing, or may be stored in a processed form suitable for training and testing an object detection model. For example, only a partial area having a preset resolution including the observation area OA among the images 200 may be stored in the DB 130 .

In the DB 130 , as metadata of the images 200 , accurate location information and shooting date information of a region appearing in each of the images 200 may be connected to each of the images 200 and stored. The location information may be location coordinates of four corners of a region appearing in each of the images 200 . According to another example, the location information may include coordinates and directions of a central location of a region appearing in each of the images 200 , and a horizontal length and a vertical length of the corresponding region.

According to another example, the position of the sun and pose information of the camera at the corresponding shooting date and time may be stored in the DB 130 instead of the shooting date and time of each of the images 200 .

The controller 110 may determine the observation area OA and a plurality of candidate areas (CRO of FIG. 6 ) within the observation area OA. The controller 110 may label location information on each of the plurality of images 200 in which at least a part of the observation area OA is displayed. The controller 110 trains an artificial neural network based on a convolutional neural network based on a plurality of images 200 labeled with location information to search each of a plurality of candidate regions CRO in each of the plurality of images 200 . can do. The controller 110 may evaluate the searched performance of each of the plurality of candidate regions CRO based on the trained artificial neural network. The controller 110 may determine at least some of the plurality of candidate areas CRO as the singular areas SRO based on the search target performance of each of the plurality of candidate areas CRO.

According to another exemplary embodiment, the controller 110 may generate an onboard map 300 of FIG. 5 including the singular areas SRO in the observation area OA.

According to another embodiment, the controller 110 may determine a landing point LS at which the lander 10 will land and a landing trajectory LT for landing at the landing point LS. According to an example, the observation area OA may be determined according to the landing trajectory LT. According to another example, the landing trajectory LT or the landing date may be determined based on the singular areas SRO in the observation area OA.

The control unit 110 will be described in more detail with reference to FIG. 8 .

Referring to FIG. 8 , the controller 110 may include a data learner 111 , a data recognizer 112 , a performance map generator 113 , and an onboard map generator 114 . The memory 120 may store an object detection model 121 based on a convolutional neural network.

The data learner 111 may learn a criterion for searching each of the candidate regions in each of the training images. The data learning unit 111 may learn a criterion regarding which data to use in order to search for each of the candidate regions in each of the training images.

The data learner 111 may train the object detection model 121 using training images labeled with location information. The data learning unit 111 performs machine learning on the object detection model 121 using training images labeled with location information to search for each candidate region in each of the training images input to the object detection model 121 . standards can be learned.

According to an embodiment, the data learning unit 111 may select training images from images stored in the DB 130 . The images stored in the DB 130 may be images obtained by at least partially capturing the observation area OA. The images may include at least some of satellite images, surface-captured images, and synthetic images.

The data recognizer 112 may search for each of the candidate regions in each of the test images. The data recognizer 112 may search for each of the candidate regions in each of the test images by using the pre-trained object detection model 121 .

The data recognizer 112 may output estimated image coordinates of each of the candidate regions in each of the test images. In each of the test images, the actual position of each of the candidate regions and the estimated image coordinates may be compared.

The data recognizer 112 may search for candidate regions in each of the test images by inputting test images to the object detection model 121 trained by the data learner 111 .

The data recognition unit 112 may post-process the output of the object detection model 121 to more intuitively understand the searched performance of each of the candidate regions.

A result value output by the object detection model 121 after receiving the test images may be used to update the object detection model 121 .

At least one of the data learning unit 111 and the data recognition unit 112 may be manufactured in the form of at least one hardware chip and mounted on a device. For example, at least one of the data learning unit 111 and the data recognition unit 112 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (eg, CPU). Alternatively, it may be manufactured as a part of an application processor) or a graphics-only processor (eg, GPU) and mounted on the various devices described above.

In this case, the data learning unit 111 and the data recognition unit 112 may be mounted on one device or may be mounted on separate devices, respectively. For example, one of the data learning unit 111 and the data recognition unit 112 may be included in the computing device 100 , and the other may be included in another computing device connected to the computing device 100 through a network. The data learning unit 111 and the data recognition unit 112 may provide the learning model information built by the data learning unit 111 to the data recognition unit 112 through a wired or wireless connection, and the data recognition unit 112 ) may be provided to the data learning unit 111 as additional learning data.

Meanwhile, at least one of the data learning unit 111 and the data recognition unit 112 may be implemented as a software module. When at least one of the data learning unit 111 and the data recognition unit 112 is implemented as a software module (or a program module including instructions), the software module is a computer-readable, non-transitory, non-transitory It may be stored in a readable recording medium (non-transitory computer readable media). Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. A part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

The performance map generator 113 may evaluate the searched performance of each of the candidate regions using the object detection model 121 . The performance map generator 113 may generate a classification performance map and an accuracy performance map of each of the candidate areas in order to indicate the searched performance of each of the candidate areas.

The onboard map generator 114 may determine specific regions having high search performance based on the search target performance of each of the candidate regions. The onboard map generator 114 may generate an onboard map including specific regions.

Before describing the performance map generating unit 113 and the onboard map generating unit 114 in detail, the data learning unit 111 and the data recognition unit 112 will be described in more detail with reference to FIGS. 9 and 10 .

9 is a block diagram of the data learning unit 111 according to an embodiment of the present invention.

Referring to FIG. 9 , the data learning unit 111 includes a data acquiring unit 111-1, a preprocessing unit 111-2, a training data selection unit 111-3, a model learning unit 111-4, and a model. The evaluation unit 111 - 5 may be included.

The data acquisition unit 111-1 may acquire data necessary to search for each of the candidate regions in each of the images. The data acquisition unit 111-1 may acquire training images from images showing a part of the observation area stored in the DB 130 . The data acquisition unit 111-1 may acquire location information stored in relation to each of the training images. The data acquisition unit 111-1 may acquire position information of the sun and/or pose information of the camera stored in relation to each of the training images.

The preprocessor 111 - 2 may preprocess the acquired data so that data necessary for searching each candidate region in each of the images may be used. The preprocessing unit 111-2 may process the obtained data into a preset format so that the model learning unit 111-4 trains the object detection model 121 for searching each candidate region in each of the images. have.

According to an embodiment, the preprocessor 111 - 2 may enlarge or reduce the training images. The preprocessor 111 - 2 may reduce or enlarge the magnifications of the training images to the same preset magnification. The landing site at which the lander will land and the landing trajectory for landing at the landing site may be determined. As the lander flies along the landing orbit, a camera for photographing the observation area and a location for photographing the observation area may also be determined.

Since the camera has the maximum spatial resolution that can be photographed and the distance from the position where the observation area is to be photographed to the observation area is determined, the magnification of the surface image generated by photographing the observation area from the lander is predetermined. The preprocessor 112 - 2 may reduce or enlarge the training image so that the magnification of the training image is the same as the magnification of the ground image. The magnification of the image refers to the size (eg, the length or area of a square) of the observation area indicated by one pixel of the image.

By matching the magnification of the training images with the magnification of the ground image, the accuracy of the object detection model 121 may be guaranteed even in the lander.

According to an embodiment, the preprocessor 111 - 2 may label each of the training images with location information. The location information may include location coordinates of a photographing area displayed in each of the training images. The preprocessor 111 - 2 may extract image coordinates of each of the candidate regions from each of the training images based on the coordinates of the location of the shooting region displayed in each of the training images. The preprocessor 111 - 2 may label each training image with image coordinates of each of the candidate regions extracted from each training image.

According to an embodiment, when determining candidate regions in the observation region, image coordinates of candidate regions in each of the training images may be stored in the DB 130 in association with each of the training images, and the preprocessor 111-2 may be used. may label the image coordinates of each of the candidate regions for each training image.

According to an embodiment, the preprocessor 111 - 2 may label the position of the sun with respect to the observation area corresponding to each of the training images and the pose of the camera for the observation area on each of the training images. According to an example, when the training image is a satellite image of the observation area, the position of the sun with respect to the observation area at the time of photographing the observation area, and the pose of the camera photographing the observation area are labeled in each of the training images. can According to another example, when the training image is a synthetic image generated by rendering, the position of the sun with respect to the observation region and the pose of the camera with respect to the observation region input to the rendering model may be labeled in each of the training images.

According to an example, in the case of a first training image among training images, an identification number and image coordinates of each of the candidate regions in the first training image may be labeled in the first training image. In addition, the position of the sun with respect to the observation area and the pose of the camera with respect to the observation area corresponding to the first training image may be labeled in the first training image.

When the candidate regions have different sizes and different shapes as shown in FIG. 6A , parameters for specifying each of the candidate regions may be labeled in each of the training images together with an identification number of the candidate regions. As shown in FIGS. 6B and 6C , when the candidate regions all have the same shape and the same size, the size and shape of each of the training images may not be labeled.

The labeling function of the preprocessor 111 - 2 may be automatically performed by the control unit 110 . The labeling function of the preprocessor 111-2 may be performed by the controller 110 through a user's manual operation.

The learning data selection unit 111-3 may select data necessary for learning from among the pre-processed data. The selected data may be provided to the model learning unit 111-4. The learning data selector 111-3 may select data required for learning from among the preprocessed data according to a preset criterion for searching each of the candidate regions in each of the training images. The preset reference may be prepared by learning by the model learning unit 111-4, which will be described later.

The model learning unit 111-4 may learn a criterion for searching each of the candidate regions in each of the training images. The model learning unit 111-4 may learn a criterion for which training data should be used in order to search for each of the candidate regions in each of the training images.

The object detection model 121 may be pre-built based on an artificial neural network. For example, the object detection model 121 may be pre-built by receiving basic learning data (eg, sample data, etc.).

The object detection model 121 may be constructed in consideration of the field of application of the recognition model, the purpose of learning, or the computer performance of the device. The object detection model 121 is, for example, a learning model based on a neural network, for example, a Deep Neural Network (DNN), a Convolution Neural Network (CNN), a Recurrent Neural Network (RNN), or a Bidirectional Recurrent (BRDNN). A model such as a Deep Neural Network) may be used, but is not limited thereto.

For the object detection model 121, algorithms such as YOLO9000, Faster R-CNN, and SSD may be used, but is not limited thereto.

The model learning unit 111-4 may train the object detection model 121 using, for example, a learning algorithm including error back-propagation or gradient descent. The model learning unit 111-4 may train the object detection model 121 through supervised learning using, for example, learning data as an input value. The model learning unit 111-4 may train the object detection model 121 through, for example, reinforcement learning using feedback on whether a result of searching each candidate region in each of the training images according to learning is correct. can

The model learning unit 111-4 may define a loss function based on the result of the object detection model 121 receiving the training image and location information labeled on the training image. When the model learning unit 111-4 inputs the first training image to the object detection model 121, candidate regions may be searched for in the first training image.

The result of the object detection model 121 may include an estimated identification number, an estimated image coordinate, and an estimated size of each of the candidate regions searched for in the first training image. The result of the object detection model 121 may include estimation accuracy of each of the candidate regions searched for in the first training image. The model learning unit 111-4 may output an estimation result having an estimation accuracy higher than a preset estimation accuracy. The model learning unit 111-4 may output a preset number of estimation results based on estimation accuracy.

The loss function may be defined based on a difference between the position information labeled on the training image and the estimation result. For example, a loss function may be defined based on a position error and a size error of each of the candidate regions. The position error may be a difference between the actual image coordinates labeled in each of the candidate regions and the image coordinates estimated for each of the candidate regions. The size error may be a difference between an actual size determined for each of the candidate regions and a size estimated for each of the candidate regions. When the candidate regions of FIG. 6C are used, the size error may be a difference between an actual radius of the candidate regions and a radius estimated for each of the candidate regions.

According to an embodiment, when the first training image is input to the object detection model 121 , the estimation result of the object detection model 121 may include estimated image coordinates of candidate regions in the first training image. . The model learning unit 111-4 trains the object detection model 121 such that a difference between the image coordinates of the candidate regions labeled in the first training image and the estimated image coordinates of the candidate regions in the first training image is minimized. can do.

Also, the estimation result of the object detection model 121 may include estimated sizes of candidate regions in the first training image. The model learner 111-4 may train the object detection model 121 to minimize a difference between the sizes of the candidate regions and the estimated sizes of the candidate regions in the first training image.

The model learning unit 111-4 may repeat the learning of the object detection model 121 using the training images so that the loss function value is smaller than a preset reference value.

When the object detection model 121 is learned, the model learning unit 111-4 may store the learned object detection model 121 . In this case, the model learning unit 111-4 may store the object detection model 121 in the memory 120 of the computing device 100 .

The memory 120 storing the object detection model 121 may also store, for example, commands or data related to at least one other component of the device. In addition, the memory 120 may store software and/or programs. A program may include, for example, a kernel, middleware, an application programming interface (API) and/or an application program (or "application"), and the like.

The model evaluation unit 111-5 inputs verification data to the object detection model 121, and when the recognition result output from the verification data does not satisfy a predetermined criterion, causes the model learning unit 111-4 to learn again. can make it In this case, the verification data may be preset data for evaluating the learning model. The verification data may be verification images selected from images stored in the DB 130 and labeled with location information.

For example, the model evaluator 111 - 5 may calculate a loss function value based on an estimation result of the learned object detection model 121 with respect to the verification data. The model evaluator 111 - 5 may evaluate the learned object detection model 121 as unsuitable when the loss function value exceeds a preset threshold.

On the other hand, the data acquisition unit 111-1, the preprocessor 111-2, the training data selection unit 111-3, the model learning unit 111-4, and the model evaluation unit 111 in the data learning unit 111. At least one of -5) may be manufactured in the form of at least one hardware chip and mounted on a device. For example, at least one of the data acquisition unit 111-1, the preprocessor 111-2, the training data selection unit 111-3, the model learning unit 111-4, and the model evaluation unit 111-5 One may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or it may be manufactured as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) as described above. It may be mounted on various devices.

In addition, the data acquisition unit 111-1, the preprocessor 111-2, the training data selection unit 111-3, the model learning unit 111-4, and the model evaluation unit 111-5 are one device. It may be mounted on the , or may be respectively mounted on separate devices. For example, some of the data acquisition unit 111-1, the preprocessor 111-2, the training data selection unit 111-3, the model learning unit 111-4, and the model evaluation unit 111-5 may be included in the device, and the rest may be included in the server.

In addition, at least one of the data acquisition unit 111-1, the preprocessor 111-2, the training data selection unit 111-3, the model learning unit 111-4, and the model evaluation unit 111-5 is It can be implemented as a software module. At least one of the data acquisition unit 111-1, the preprocessor 111-2, the training data selection unit 111-3, the model learning unit 111-4, and the model evaluation unit 111-5 is a software module When implemented as (or, a program module including instructions), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. A part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

10 is a block diagram of the data recognition unit 112 according to an embodiment of the present invention.

Referring to FIG. 10 , the data recognition unit 112 according to some embodiments includes a data acquisition unit 112-1, a preprocessor 112-2, a recognition data selection unit 112-3, and a recognition result providing unit ( 112-4) and a model updater 112-5.

The data acquisition unit 112-1 may acquire data necessary to search for each of the candidate regions in each of the images. The data acquisition unit 112-1 may acquire test images from images showing a part of the observation area stored in the DB 130 . The data acquisition unit 112-1 may acquire location information stored in relation to each of the test images. The data acquisition unit 112-1 may acquire position information of the sun and/or pose information of the camera stored in relation to each of the test images.

The preprocessor 112 - 2 may preprocess the obtained data so that data necessary for searching each candidate region in each of the test images may be used. The preprocessing unit 112-2 may process the acquired data into a preset format so that the recognition result providing unit 112-4 may search for each candidate region in each of the images using the acquired data.

According to an embodiment, the preprocessor 112 - 2 may enlarge or reduce the test images. The preprocessor 112 - 2 may reduce or enlarge the magnifications of the test images to the same preset magnification. The preprocessor 112 - 2 may reduce or enlarge the test image so that the magnification of the test image is the same as the magnification of the ground image.

According to an embodiment, the preprocessor 112 - 2 may label location information on each of the test images. The location information may include location coordinates of a photographing area displayed in each of the test images. The preprocessor 112 - 2 may extract image coordinates of each of the candidate regions from each of the test images based on the coordinates of the location of the photographing area displayed in each of the test images. The preprocessor 112 - 2 may label each test image with image coordinates of each of the candidate regions extracted from each test image.

According to an embodiment, when determining candidate regions within the observation region, image coordinates of the candidate regions in each of the test images may be stored in the DB 130 in association with each of the test images, and the preprocessor 112 - 2 ) may label the image coordinates of each of the candidate regions for each test image.

According to an embodiment, the preprocessor 112 - 2 may label the position of the sun with respect to the observation area corresponding to each of the test images and the pose of the camera with respect to the observation area on each of the test images.

According to an example, in the case of a first test image among the test images, an identification number and image coordinates of each of the candidate regions in the first test image may be labeled on the first test image. In addition, the position of the sun with respect to the observation area and the pose of the camera with respect to the observation area corresponding to the first test image may be labeled on the first test image.

The labeling function of the preprocessor 112 - 2 may be automatically performed by the control unit 110 . The labeling function of the preprocessor 112-2 may be performed by the controller 110 through a user's manual operation.

The recognition data selection unit 112 - 3 may select data required to search for each of the candidate regions in each of the test images from among the pre-processed data. The recognition data selector 112 - 3 may select some or all of the preprocessed data according to a preset criterion for searching each candidate region in each of the test images. In addition, the recognition data selection unit 112-3 may select data according to a criterion set by the model learning unit 111-4 learning.

The recognition result providing unit 112 - 4 may search for each of the candidate regions in each of the test images by applying the selected data to the object detection model 121 . The recognition result providing unit 112 - 4 may provide a recognition result according to the purpose of data recognition. The recognition result providing unit 112 - 4 may apply the selected data to the object detection model 121 by using the data selected by the recognition data selecting unit 112 - 3 as an input value. Also, the recognition result may be determined by the object detection model 121 . For example, the identification number and location of each of the candidate regions searched in each of the test images may be estimated, and the estimation result may be a text, an image, or a command (eg, an application execution command, a module function execution command, etc.) ) may be provided.

According to an embodiment, when the recognition result providing unit 112 - 4 inputs the first test image to the object detection model 121 , the estimation result of the object detection model 121 is a candidate region in the first test image. may include the estimated image coordinates of The estimated image coordinates of the candidate region may mean the center image coordinates of the candidate region. Also, the estimation result of the object detection model 121 may include estimated sizes of candidate regions in the first training image.

For example, the first candidate region may be searched for in the first test image, and in this case, the estimation result of the object detection model 121 may include estimated image coordinates of the first candidate region in the first test image. The first candidate area found in the first test image may or may not be the first candidate area of the actual observation area. That is, the estimated image coordinates of the first candidate area may or may not correspond to the first candidate area in the actual first observation area.

Otherwise, the estimated image coordinates of the first candidate area may correspond to other candidate areas within the observation area or may correspond to a background area within the first observation area. In this case, the estimation result of the object detection model 121 corresponds to an erroneous estimation.

The second candidate region may not be searched for in the first test image, and in this case, information on the second candidate region may not be included in the estimation result of the object detection model 121 . The third candidate image may be searched for in two positions in the first test image. In this case, the estimation result of the object detection model 121 is the first estimated image coordinates and the second estimated image coordinates of the third candidate region in the first test image. It may include estimated image coordinates.

Since only one third candidate region actually exists in the observation region, at least one of two estimation results for the third candidate region of the object detection model 121 corresponds to an erroneous estimation.

The model updater 112 - 5 may update the object detection model 121 based on the evaluation of the recognition result provided by the recognition result provider 112 - 4 . For example, the model updating unit 112-5 provides the recognition result provided by the recognition result providing unit 112-4 to the model learning unit 111-4, so that the model learning unit 111-4 is The object detection model 121 may be updated.

On the other hand, in the data recognition unit 112, the data acquisition unit 112-1, the preprocessor 112-2, the recognition data selection unit 112-3, the recognition result providing unit 112-4 and the model update unit ( At least one of 112-5) may be manufactured in the form of at least one hardware chip and mounted in a device. For example, among the data acquisition unit 112-1, the preprocessor 112-2, the recognition data selection unit 112-3, the recognition result providing unit 112-4, and the model update unit 112-5 At least one may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU). It may be mounted on one various device.

In addition, the data acquisition unit 112-1, the preprocessor 112-2, the recognition data selection unit 112-3, the recognition result providing unit 112-4, and the model update unit 112-5 are one It may be mounted on an electronic device, or may be respectively mounted on separate electronic devices. For example, among the data acquisition unit 112-1, the preprocessor 112-2, the recognition data selection unit 112-3, the recognition result providing unit 112-4, and the model update unit 112-5 Some may be included in the computing device 100 , and some may be included in the server.

In addition, at least one of the data acquisition unit 112-1, the preprocessor 112-2, the recognition data selection unit 112-3, the recognition result providing unit 112-4, and the model update unit 112-5 may be implemented as a software module. At least one of the data acquisition unit 112-1, the preprocessor 112-2, the recognition data selection unit 112-3, the recognition result providing unit 112-4, and the model update unit 112-5 is software. When implemented as a module (or a program module including instructions), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. A part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

Hereinafter, a configuration for generating a performance map and an onboard map by using the estimation results for each of the test images of the object detection model 121 provided by the recognition result providing unit 112 - 4 will be described.

Referring back to FIG. 8 , the performance map generator 113 may evaluate the searched performance of each of the candidate regions by using the estimation result of the object detection model 121 . The searched performance is evaluated for each of the candidate regions.

For example, the searched performance of the first candidate region may be evaluated based on the classification performance and accuracy performance of the first candidate region. The classification performance of the first candidate region is calculated using an estimation result obtained when test images are input to the object detection model 121 . The estimation result includes estimated image coordinates of candidate regions in each of the test images.

The classification performance of the first candidate regions may be calculated based on _{an F 1} score calculated based on estimated image coordinates of the first candidate regions. The _{higher the F 1} score, the better the classification performance.

The F ₁ score is calculated based on the Precision and Recall values. For example, the F ₁ score may be calculated as a harmonic average of the reciprocal of a precision value and a recall value.

The precision value can be calculated as the value (TP/ (TP + FP)) obtained by dividing the true positive (TP) by the sum (TP + FP) of the true positive (TP) and false positive (FP). . The recall value can be calculated as a value (TP/ (TP + FN)) obtained by dividing True Positive (TP) by the sum (TP + FN) of True Positive (TP) and False Negative (FN). .

According to an embodiment, true positivity means that the estimated position of the candidate area estimated by the object detection model 121 is included in the corresponding candidate area of the actual observation area. For example, when the first candidate area is found in the test image, the estimated position of the first candidate area is located in the first candidate area of the actual observation area.

False positive means that the estimated position of the candidate area estimated by the object detection model 121 is included in another candidate area of the actual observation area. For example, when the first candidate area is found in the test image, the estimated position of the first candidate area is located in a candidate area other than the first candidate area of the actual observation area (eg, the second candidate area).

The false negative means that the estimated position of the candidate region estimated by the object detection model 121 is included in the background region of the actual observation region. For example, when the first candidate area is found in the test image, the estimated position of the first candidate area is located in the background area of the actual observation area.

According to another embodiment, the classification performance of the first candidate region is the number of true positives (TP) of the test images in which the estimated image coordinates of the first candidate region in each of the test images are included in the actual first candidate region in the observation region; The number of false positives (FP) of the test images in which the estimated image coordinates of the first candidate region in each of the test images are not included in the actual first candidate region in the observation region, and the estimation of the first candidate region in each of the test images The image coordinates may be determined based on the number of false voices (FN) of the non-existent test image. For example, the classification performance of the first candidate region may be calculated by 2 TP / ( 2 TP + FN + FP).

The accuracy performance of the first candidate regions may be calculated based on estimated image coordinates of the first candidate regions. According to an embodiment, the accuracy performance of the first candidate region may be calculated based on the difference in position between the image coordinates of the first candidate region labeled in each test image and the estimated image coordinates of the first candidate region in each test image. can The accuracy performance of the first candidate region may be calculated as a root mean square error of position differences calculated for each of the test images. The lower the accuracy performance value, the better the accuracy performance.

According to another exemplary embodiment, the accuracy performance of the first candidate region is determined by determining that the estimated position of the first candidate region is the actual position of the first candidate region when the estimated image coordinates of the first candidate region are included in the actual first candidate region of the observation region. It can indicate how far apart the location is. If the estimated image coordinates of the first candidate region are not included in the actual first candidate region of the observation region, the position difference between the estimated position of the first candidate region and the actual position of the first candidate region is the accuracy performance of the first candidate region. may not affect

The performance map generator 113 may generate a classification performance map and an accuracy performance map of each of the candidate areas in order to indicate the searched performance of each of the candidate areas. The classification performance map and the accuracy performance map may be referred to as a classification performance graph and an accuracy performance graph, respectively.

A classification performance map and an accuracy performance graph are generated for each of the candidate regions. The classification performance map and the accuracy performance map of each of the candidate regions show that the classification performance and accuracy performance of each of the candidate regions according to the change in the azimuth and elevation of the sun relative to the observation region labeled in each of the test images. shows how it changes.

Each of the test images may be labeled with the position (azimuth and altitude) of the sun with respect to the observation area. For example, the classification performance map of the first candidate region calculates the classification performance value of the first candidate region in each of the test images, and the x-coordinate and y-coordinate determined by the azimuth and elevation of the sun labeled in each of the test images. It may be a graph in which the classification performance value of the first candidate region is displayed at the position of . In this graph, the x-axis corresponds to the azimuth of the sun and the y-axis corresponds to the elevation of the sun. An area in which a classification performance value is not displayed in the classification performance map may be displayed through interpolation using surrounding values.

The accuracy performance map of the first candidate region calculates the accuracy performance value of the first candidate region in each of the test images, and the x-coordinate and y-coordinate positions determined by the azimuth and elevation of the sun labeled in each of the test images It may be a graph in which the accuracy performance value of the first candidate region is displayed. In this graph, the x-axis corresponds to the azimuth of the sun and the y-axis corresponds to the elevation of the sun. An area in which the accuracy performance value is not displayed in the accuracy performance map may be displayed through interpolation using surrounding values.

The onboard map generator 114 selects candidate regions having high search performance based on the classification performance map and the accuracy performance map of each of the candidate regions, and when the number of candidate regions is equal to or greater than a preset reference number, selects the selected candidate regions. It can be determined by specific regions.

Through the classification performance map and the accuracy performance map, an area in which a corresponding candidate area can be searched well with respect to the position of the sun may be determined. An area in which classification performance and accuracy performance are equal to or greater than the reference value may be an area in which the corresponding candidate area is searched well. Candidate regions in which this region is wide may be determined as singular regions.

Because the moon has no atmosphere, there is little change in its topography. When an onboard map including the determined specific regions is generated, the generated onboard map may be used semi-permanently.

The position of the sun with respect to the observation area is determined by the ephemeris. The position (azimuth and elevation) of the sun determined by the ephemeris may be further displayed in the classification performance map and the accuracy performance map. When the landing date is determined, candidate regions having high classification performance and high accuracy performance on the corresponding date may be determined.

When the classification performance value is higher than the preset classification reference value, it may be understood that the classification performance is high. When the accuracy performance value is lower than the preset accuracy reference value, it may be understood that the accuracy performance is high.

Among the candidate regions, candidate regions having a classification performance value greater than a preset classification reference value and an accuracy performance value lower than a preset accuracy reference value may be selected. The number of selected candidate regions may be compared with a preset reference number, and when the number of selected candidate regions is equal to or greater than the preset reference number, the selected candidate regions may be determined as singular regions. The onboard map generator 114 may generate an onboard map including the determined specific regions.

If the number of the selected candidate areas is smaller than the preset reference number, it is determined that no singular areas are detected according to the previously determined configuration of the candidate areas of the observation area, and the observation area is changed or the configuration of the candidate areas is changed. can

A preliminary landing date may be determined. Candidate regions having high classification performance and accuracy performance of the corresponding preliminary landing date may be selected. When the number of selected candidate regions is equal to or greater than a preset reference number, the selected candidate regions may be determined as preliminary singular regions. The onboard map generator 114 may generate a preliminary onboard map including the determined preliminary specific regions.

The onboard map including the singular regions generated by the onboard map generator 114 may be loaded in the memory of the lander of FIGS. 1 to 4 and used for optical navigation for accurately flying to the landing site.

The various embodiments described above are exemplary, and do not need to be performed independently of each other. Embodiments described in this specification may be implemented in a form in combination with each other.

The various embodiments described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium may be to continuously store the program executable by the computer, or to temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers.

In this specification, "unit", "module", etc. may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor. For example, “part”, “module” and the like refer to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, It may be implemented by procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays and variables.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims

A method performed by a computing device, comprising:

determining an observation area and a plurality of candidate areas within the observation area;

labeling location information on each of a plurality of images in which at least a part of the observation area appears;

training a convolutional neural network-based artificial neural network based on the plurality of images labeled with the location information to search each of the plurality of candidate regions in each of the plurality of images;

evaluating the searched performance of each of the plurality of candidate regions based on the trained artificial neural network; and

and determining at least some of the plurality of candidate regions as singular regions based on the searched performance of each of the plurality of candidate regions.
According to claim 1,

The method for determining a singular region, characterized in that the plurality of candidate regions are distributed so as not to overlap each other within the observation region.
According to claim 1,

The plurality of candidate regions have a preset shape and are arranged adjacent to each other in the observation region.
According to claim 1,

The plurality of candidate regions are circular having a first diameter and are arranged adjacent to each other within the observation region.
According to claim 1,

The plurality of images of the observation area includes satellite images photographed by at least one artificial satellite flying along an orbit having a period on the observation area.
According to claim 1,

A plurality of images of the observation area is a singular area determination method, characterized in that it includes images captured by a flying vehicle flying over the observation area.
According to claim 1,

The plurality of images obtained by photographing the observation area includes images obtained by photographing the observation area on Earth.
According to claim 1,

The plurality of images of the observation area may include synthesized images generated based on the modeled topography of the observation area, the position of the sun with respect to the observation area, and a pose of a camera photographing the observation area. A method for determining a specific region.
According to claim 1,

determining a landing point on which the lander will land and a landing trajectory for landing at the landing site;

The method for determining a singular area, characterized in that the observation area is determined according to the landing trajectory.
10. The method of claim 9,

The method further comprising enlarging or reducing the plurality of images based on the altitude of the landing trajectory on the observation area and the spatial resolution of a camera mounted on the lander.
According to claim 1,

The location information includes location coordinates of the shooting area shown in each of the plurality of images,

The method of claim 1, wherein the image coordinates of each of the plurality of candidate regions in the respective images are extracted based on the coordinates of the location of the photographing region labeled in the respective images.
According to claim 1,

The location information comprises image coordinates of the plurality of candidate regions in each image.
According to claim 1,

The labeling includes additionally labeling each of the plurality of images with a position of the sun with respect to the observation area at a viewpoint corresponding to each image and a pose of a camera with respect to the observation area. A method for determining a specific region.
According to claim 1,

Training the artificial neural network comprises:

inputting a first image from among the images labeled with the location information to the artificial neural network;

receiving, as an output of the artificial neural network, estimated image coordinates of the plurality of candidate regions in the first image; and

training the artificial neural network to minimize a difference between the image coordinates of the plurality of candidate regions labeled in the first image and the estimated image coordinates of the plurality of candidate regions in the first image A method for determining a specific region characterized.
15. The method of claim 14,

Training the artificial neural network comprises:

receiving, as an output of the artificial neural network, estimated sizes of the plurality of candidate regions in the first image; and

and training the artificial neural network to minimize a difference between the sizes of the plurality of candidate regions and the estimated sizes of the plurality of candidate regions in the first image.
According to claim 1,

Evaluating the searched performance of each of the plurality of candidate areas includes evaluating the searched performance of a first candidate area among the plurality of candidate areas,

Evaluating the search target performance of the first candidate area includes:

inputting each of a plurality of test images in which at least a part of the observation area appears to the trained artificial neural network;

receiving, as an output of the trained artificial neural network, estimated image coordinates of the plurality of candidate regions in each of the plurality of test images; and

calculating an F 1 score based on an output of the trained artificial neural network, and determining classification performance of the first candidate region based on the calculated F 1 score.
17. The method of claim 16,

Evaluating the search target performance of the first candidate area includes:

Determining accuracy performance of the first candidate region based on a position difference between the image coordinates of the first candidate region labeled in each test image and the estimated image coordinates of the first candidate region in each test image A method for determining a specific region, comprising the steps of.
18. The method of claim 17,

Evaluating the search target performance of the first candidate area includes:

Based on the position of the sun with respect to the observation area labeled in each of the plurality of test images, the classification performance and accuracy performance of the first candidate area according to the azimuth and altitude of the sun are respectively classified into a classification performance graph and an accuracy performance graph. A method for determining a specific region further comprising the step of displaying.
19. The method of claim 18,

The determining of at least some of the plurality of candidate regions as singular regions includes determining the singular regions based on a classification performance graph and an accuracy performance graph for each of the plurality of candidate regions. A method for determining a specific region.
19. The method of claim 18,

determining a landing point on which the lander will land and a landing trajectory for landing at the landing site;

Determining at least some of the plurality of candidate regions as singular regions includes:

determining at least some of the plurality of candidate areas as first singular areas based on a classification performance graph and an accuracy performance graph for each of the plurality of candidate areas, and a first landing date and time of the lander;

generating a first onboard map including the first singular regions within the viewing area;

determining at least some of the plurality of candidate areas as second singular areas based on a classification performance graph and an accuracy performance graph for each of the plurality of candidate areas, and a second landing date and time of the lander; and

and generating a second onboard map comprising the second singular regions within the observation region.
A computer program stored on a medium for executing the method of any one of claims 1 to 20 using a computing device.
a memory for storing a plurality of images in which at least a part of an observation area is displayed, and an artificial neural network based on a convolutional neural network; and

determining the observation area and a plurality of candidate areas within the observation area;

Labeling location information on each of the plurality of images,

training the artificial neural network based on the plurality of images labeled with the location information to search each of the plurality of candidate regions in each of the plurality of images,

Evaluating the searched performance of each of the plurality of candidate regions based on the trained artificial neural network,

determining at least some of the plurality of candidate regions as singular regions based on the searched performance of each of the plurality of candidate regions;

at least one processor configured to generate the onboard map including the singular regions within the viewing area.
A method for determining the orientation of a lander by a processor of the lander, the method comprising:

generating an observation area image by photographing a preset observation area at a preset altitude while flying according to a landing trajectory;

searching the observation area image for singular areas within the observation area of an onboard map stored in a memory; and

determining the direction of the probe based on a difference between the arrangement directions of the singular regions discovered in the observation area image and the arrangement directions of the singular regions of the onboard map;

The onboard map is

determining an observation area and a plurality of candidate areas within the observation area;

labeling location information on each of a plurality of images in which at least a part of the observation area appears;

training a convolutional neural network-based artificial neural network based on the plurality of images labeled with the location information to search each of the plurality of candidate regions in each of the plurality of images;

evaluating the searched performance of each of the plurality of candidate regions based on the trained artificial neural network;

determining at least some of the plurality of candidate areas as the singular areas based on the searched performance of each of the plurality of candidate areas; and

and generating by a computing device performing the step of generating the onboard map including the singular areas within the observation area.