JP6681965B2 - Apparatus and method for extracting learning target image for autonomous driving - Google Patents

Description

  The present invention relates to a learning target image extraction device and method for autonomous driving.

  In recent years, the trend of artificial intelligence (AI) is technically changing from voice and text recognition to video image recognition. Video image recognition can be used for controlling autonomous vehicles, analyzing black box images, CCTV images, and interpreting medical images, and has a great ripple effect in various industries. Among them, in the research and development of level 4 autonomous vehicles, it is judged that 160 billion km of video image data is required worldwide and 100 million km of image is required for each city. . In addition, recognition of video images requires a separate processing operation to use the collected video image data for intelligence of the images, but the processing operations are performed manually based on human cognitive ability, It takes a lot of time and cost.

Korean Published Patent Publication No. 10-2004-0026825

  Therefore, the present invention has been made in view of the above circumstances, and an object thereof is to provide a learning target image extraction device and method for autonomous traveling.

  Problems to be solved by the present invention are not limited to the problems described above, and other problems not mentioned above should be clearly understandable by a person skilled in the art from the following description.

  An apparatus and method for extracting a learning target image for autonomous driving according to an aspect of the present invention for solving the above-described problems include an original moving image receiving step of receiving an original road driving image and metadata of the original road driving image. A primary sampling step of determining an extraction number based on the first road image and extracting the determined number of frame images from the original road running image; and each of the determined number of frame images includes a first learning target object. Whether or not there is a second sampling step of extracting one or more learning target frame images from the determined number of frame images based on the movement amount of the first learning target object. The metadata of the road driving image is related to the vehicle associated with the original road driving image. GPS information, including one or more of the imaging time of the moving speed information and the original road traveling image of the vehicle.

  In one embodiment, the primary sampling step further includes a congestion variable calculation step of calculating a congestion variable of an area associated with the original road driving image, and further including a congestion variable calculation step of calculating a congestion variable of the area associated with the original road traveling image. Based on this, the number of extractions is determined.

  In one embodiment, the crowded variable calculating step uses one or more of demographic data or vehicle statistical data of a region associated with the original road driving image to collect a crowded variable of the region associated with the original road driving image. To calculate.

  In one embodiment, the secondary sampling step uses an object detection algorithm to generate one or more first learning targets of one or more predetermined types on each of the determined number of frame images. A first learning target object detecting step of detecting an object, a detected position of the one or more first learning target objects on the n-th (n is a natural number of 2 or more) frame image, and the n-th frame image The position of the one or more first learning target objects detected on the immediately preceding (n−1) th frame image is compared, and the detected one or more first learning targets on the nth frame image are compared. The method includes a first learning target object movement amount measuring step of measuring a movement amount of each object.

  In one embodiment, the secondary sampling step further includes a background removal step of removing a background excluding the first learning target object on each of the determined number of frame images.

  In one embodiment, the method further includes a learning data set generation step of processing the one or more learning target frame images to generate a learning data set.

  In one embodiment, the learning data set generating step includes a second learning target object detecting step of detecting one or more second learning target objects of one or more predetermined types on the learning target frame image. And a learning target object annotating step of performing annotation processing on the one or more second learning target objects detected on the learning target frame image.

  In one embodiment, the learning data set generating step further includes a validity verification step of verifying the validity of the learning data set.

  A learning target image extracting device for autonomous traveling according to another aspect of the present invention for solving the above-mentioned problem is an artificial intelligence learning target image generating device for autonomous traveling, which receives an original road traveling image. A road traveling image receiving unit, a first sampling unit that determines an extraction number based on the metadata of the original road traveling image, and extracts the determined number of frame images from the original road traveling image, and the determined One or more learning target frame images from the determined number of frame images based on whether or not each of the number of frame images includes the first learning target object and the movement amount of the first learning target object. And a second sampling unit for extracting the meta data of the original road driving image. Includes one or more of the original road travel video and related GPS information of the vehicle, the imaging time of the moving speed information and the original road traveling image of the vehicle.

  Other specific details of the invention are included in the detailed description and drawings.

  According to the learning target image extraction device and method for autonomous driving of the present invention, the first sampling unit performs primary sampling of the original road traveling image based on the metadata of the original road traveling image, and therefore is performed thereafter. The cost and time required for processing the data in the secondary sampling stage and the learning data set generating stage can be remarkably reduced, and the efficiency of extracting the learning target image for autonomous traveling can be improved. Play.

  In addition, the second sampling unit sets a reference of the movement amount of the first learning target object and selects a frame image corresponding to a predetermined reference or more as a learning target frame image, thereby eliminating unnecessary frames. The image is reduced and only the required frame image can be selected.

  Further, by extracting only the core frame images as the learning target frame images through the primary sampling and the secondary sampling, it is possible to obtain the same frame images without sampling, although the number of learning target frame images is small. A learning target image for autonomous driving having a level of performance can be extracted.

  The effects of the present invention are not limited to the effects described above, and other effects that are not mentioned should be clearly understood by those of ordinary skill in the art from the following description.

3 is a schematic flowchart of a learning target image extraction method for autonomous driving according to an exemplary embodiment of the present invention. FIG. 3 is a view showing an example of a frame image extracted by the primary sampling stage of FIG. 1 according to an exemplary embodiment of the present invention. 2 is a schematic flowchart of a secondary sampling stage of FIG. 1. FIG. 3 is an exemplary diagram illustrating a method for measuring a movement amount of a first learning target object in the secondary sampling stage of FIG. 1. 2 is a schematic flowchart of the secondary sampling stage of FIG. 1 with a background removal stage added. FIG. 6 is an exemplary diagram illustrating a method for measuring a movement amount of a first learning target object using a frame image in which a background is removed in the secondary sampling stage of FIG. 1 to which a background removal stage is added. 9 is a schematic flowchart of a learning target image extraction method for autonomous traveling to which a congestion variable calculation step is added according to another exemplary embodiment of the present invention. 9 is a schematic flowchart of a learning target image extraction method for autonomous traveling to which a learning data set generation step is added according to another embodiment of the present invention. 9 is a schematic flowchart of the learning data set generation step of FIG. 8. 9 is a schematic flowchart of a learning data set generation step of FIG. 8 to which a validity verification step is added. 11 is a schematic flowchart of the validity verification stage of FIG. 10. It is a schematic block diagram of the learning target image extraction apparatus for autonomous traveling which concerns on other embodiment of this invention.

  The advantages and features of the present invention, and the manner of achieving them, will be apparent with reference to the embodiments described in detail below in connection with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and can be implemented in various forms different from each other. However, the present embodiment is provided to complete the disclosure of the present invention and to allow a person skilled in the art to which the present invention belongs to fully understand the scope of the present invention. It is defined only by the claims that follow.

  The terms used in the present specification are for the purpose of describing the embodiments and are not intended to limit the present invention. In this specification, the singular forms also include the plural forms unless otherwise specified. As used herein, “comprises” and / or “comprising” does not exclude the presence or addition of one or more other components than those mentioned. Like reference numerals refer to like elements throughout the specification, and "and / or" includes each and every combination of one or more of the referenced elements. Of course, even though "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one element from another. Therefore, it goes without saying that the first component referred to below can be the second component within the technical idea of the present invention.

  Unless defined otherwise, all terms (including technical and scientific terms) used herein have the meaning commonly understood by one of ordinary skill in the art to which this invention belongs. Also, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively, unless explicitly defined otherwise.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a schematic flowchart of a learning target image extraction method for autonomous driving according to an exemplary embodiment of the present invention.

  Referring to FIG. 1, the learning target image extraction method for autonomous driving includes an original moving image receiving step (S100), a primary sampling step (S200), and a secondary sampling step (S300).

  In operation S100, the original road running image receiving unit 610 receives the original road running image. The original road driving image is a moving image collected by a camera provided inside or outside the vehicle or on a roadway. For example, when receiving an original road driving video to generate artificial intelligence video learning data for autonomous driving of a car, the original road driving video is equipped with a camera for generating learning data for autonomous driving. It is possible to include all videos capable of generating learning data for autonomous driving, such as moving images acquired from an automobile, but the invention is not limited thereto.

  In operation S200, the first sampling unit 620 determines an extraction number based on the metadata of the original road driving image, and extracts the determined number of frame images from the original road driving image. The metadata of the original road traveling image includes at least one of the GPS information of the vehicle related to the original road traveling image, the moving speed information of the vehicle, and the shooting time of the original road traveling image.

  In an exemplary embodiment, the first sampling unit 620 may include a number corresponding to the area when the GPS information of the vehicle is included in the predetermined area based on the vehicle GPS information of the metadata of the original road driving image. Extract the frame image of. For example, the first sampling unit 620 may extract 30 frames per second when the vehicle is in the busy area A, and may extract 15 frames per second when the vehicle is in the quiet area B.

  In an exemplary embodiment, the first sampling unit 620 corresponds to the corresponding speed range when the moving speed of the vehicle is included in the predetermined speed range based on the vehicle moving speed information of the metadata of the original road driving image. Extract as many frame images as you want. For example, when the moving speed of the vehicle is 100 km / h, which is relatively high, the first sampling unit 620 extracts 30 frames per second, and when the moving speed of the vehicle is 50 km / h, which is relatively low. 15 frames can be extracted per second.

  In an exemplary embodiment, the first sampling unit 620 includes a number corresponding to the time range when the travel time of the vehicle is included in the predetermined time range based on the shooting time of the metadata of the original road driving image. Extract the frame image of. For example, the first sampling unit 620 may extract 30 frames per second in the case of 7:00 to 9:00, or 18:00 to 20:00, which is the time when the vehicle travel time is crowded, that is, the time when the vehicle travel time is crowded. In the case of 1 to 5 o'clock, which is a dawn time when the moving time is, 15 frames can be extracted per second.

  In another embodiment, if the weather information is included in the predetermined weather based on the weather information corresponding to the shooting time of the metadata of the original road driving image, the first sampling unit 620 determines that the weather is appropriate. Extract as many frame images as you want. For example, if the weather information corresponding to the shooting time is cloudy, strong wind, heavy rain, heavy snow, etc., the first sampling unit 620 extracts 30 frames per second, and if the weather information is clear, per second. 15 frames can be extracted.

  In operation S300, the second sampling unit 630 determines whether or not each of the determined number of frame images includes the first learning target object and the determined number based on the moving amount of the first learning target object. One or more frame images to be learned are extracted from the frame images. The secondary sampling step (S300) will be described in more detail with reference to FIG.

  FIG. 2 is an exemplary diagram of a frame image extracted by the primary sampling stage of FIG. 1 according to an exemplary embodiment of the present invention.

  Referring to FIG. 2, the frame image may be extracted at a rate of 60 fps (frame per second), 30 fps, or 1 fps according to the first sampling step (S200).

  In operation S200, the first sampling unit 620 determines an extraction number based on the metadata of the original road driving image, and extracts the determined number of frame images from the original road driving image. For example, if the frame rate of the original road driving image is 60 fps, the first sampling unit 620 may generate 60, 30, or 1 frame per second according to the metadata of the original road driving image. Can be extracted.

  In operation S200, the first sampling unit 620 performs primary sampling of the original road driving image based on the metadata of the original road driving image (i.e., removes unnecessary data and transmits data to the secondary sampling step). Therefore, the cost and time required to process the data in the secondary sampling step (S300) and the learning data set generating step (S500) performed later can be significantly reduced.

  FIG. 3 is a schematic flowchart of the secondary sampling stage of FIG.

  Referring to FIG. 3, the secondary sampling step (S300) includes a first learning target object detecting step (S310) and a first learning target object movement amount measuring step (S320).

  In operation S310, the second sampling unit 630 uses the object detection algorithm to detect one or more first learning target objects of one or more predetermined types on each of the determined number of frame images. To do.

  The first learning target object is an object included in the frame image. The frame image includes one or more first learning target objects, and includes one or more types of first learning target objects. For example, the first learning target object includes a person, an automobile, a bicycle, a building, a telephone pole, a motorcycle, a tree, a flower, a puppy, a cat, a road, a traffic display board, an overspeed preventing step, a road cone, a lane, and the like. However, the present invention is not limited to the above example, and includes all objects that can be distinguished as objects. Further, the type of the first learning target object includes the front surface, the rear surface, the right side surface, the left side surface, etc. of the object, but the type of each learning target object is not limited to the above example, and can be subdivided from the above example. It can be classified and can be classified as a completely different type from the above example.

  Detecting one or more objects of one or more types in the first learning target object detection is detection using an object detection algorithm, and the object detection algorithm may include an R-CNN model. However, it is not limited to this.

  In operation S320, the second sampling unit 630 determines the positions of the one or more first learning target objects detected on the n-th (n is a natural number of 2 or more) frame image and the n-th frame immediately before the n-th frame image. The positions of the one or more first learning target objects detected on the -1 frame image are compared, and the movement amount of each of the one or more first learning target objects detected on the nth frame image is measured.

  The method of extracting the learning target frame image is performed by determining whether the movement amount of the detected one or more first learning target objects is equal to or more than a predetermined reference and detecting the detected one or more first learning target objects. If the amount of movement of the first learning target object equal to or more than a predetermined number is equal to or greater than the predetermined reference, the corresponding frame image is selected as the learning target frame image. However, if the movement amount of a predetermined number or more of the first learning target objects among the detected one or more first learning target objects does not correspond to a predetermined criterion or more, the corresponding frame image is targeted for learning. Do not select as a frame image.

  By setting the reference of the moving amount of the first learning target object and selecting the frame image corresponding to the predetermined reference or more as the learning target frame image, unnecessary frame images are reduced and necessary frame images are reduced. Only images can be selected.

  Therefore, in the learning target frame image, all the extracted frame images are not selected as the learning target frame images, the moving amount of the object is small, and only the remaining frame images are excluded, which does not have a great influence on the learning. , The amount of the learning data set can be reduced in the subsequent stages by selecting the learning target frame image. Therefore, in learning the artificial intelligence module, the learning time can be shortened by using the learning data set having the reduced amount.

  In another embodiment, the second sampling unit 630 determines the presence / absence of a traffic control object from the frame image, and if the corresponding traffic control object meets a predetermined criterion, further determines the frame based on the traffic control object. Extract one or more learning target frame images from the image. The traffic control object may be a traffic light on a frame image, a traffic safety sign, a traffic safety line, a road ancillary facility, or the like, but is not limited thereto. For example, when the traffic light is detected from the frame image and the traffic light changes from red to green, the second sampling unit 630 extracts the entire frame image as a learning target frame, and when the traffic light is maintained in the red, 1 It is possible to halve the number of frame images per second and extract only part of the relevant frame images as learning target frames.

  In another embodiment, the second sampling unit 630 extracts at least one of the internal sound and the external sound of the vehicle when the original road running image is captured, and the frequencies of the internal sound and the external sound of the vehicle are predetermined. If it is within the reference range, one or more learning target frame images are extracted from the frame images based on at least one of the internal sound and the external sound of the vehicle. For example, the second sampling unit 630 extracts audio data from a black box road running image, and when a screaming voice is detected from the corresponding audio data, extracts 30 learning target frames per second. If no sound is detected, 15 learning target frames can be extracted per second.

  The first learning target object movement amount measuring step (S320) will be described in detail with reference to FIG.

  FIG. 4 is an exemplary diagram illustrating a method for measuring the movement amount of the first learning target object in the secondary sampling stage of FIG. 1.

  Referring to FIG. 4, a movement amount measuring method in the first learning object movement amount measuring step (S320) is shown. FIG. 4A shows the (n-1) th frame image 11, and FIG. 4B shows the nth frame image 12.

  The movement amount of the first learning target object is measured by comparing the position of the first learning target object 21 on the n-1th frame image 11 with the position of the first learning target object 22 on the nth frame image 12. To do.

  The first learning target object 21 on the (n−1) th frame image 11 and the first learning target object 22 on the nth frame image 12 have the same shape, and are located at the same position of the first learning target object. First select the relevant part.

  When the computer selects the specific portion as “A” in the first learning target object 21 on the (n−1) th frame image 11, the part corresponding to the same position of the first learning target object is selected as the nth frame. A portion corresponding to the same position as "A" on the first learning target object 22 on the image 12 is selected as "A '".

  The computer selects a portion corresponding to the same position of the first learning target object, places the (n-1) th frame image 11 and the nth frame image 12 on the same plane, and then performs “A” and “A ′”. Extract the coordinates for.

  After extracting the coordinates for “A” and “A ′”, the computer measures the amount of movement using the difference between the coordinates of “A” and the coordinates of “A ′”.

  FIG. 5 is a schematic flow chart of the secondary sampling stage of FIG. 1 with the addition of a background removal stage.

  Referring to FIG. 5, the background removal step (S330) is performed after the first learning target object detection step (S310).

  In operation S330, the second sampling unit 630 removes the background excluding the first learning target object on each of the determined number of frame images.

  In the background removal step (S330), the object excluding the first learning target object detected on the frame image is processed as a background, and the background portion is entirely removed. The removal of the background portion may include setting a pixel value of a region corresponding to the background to a null state or processing the pixel value with a predetermined special value. For example, the gray of the pixel in the area corresponding to the background may be processed as 0 gray or 256 gray, but is not limited thereto.

  The secondary sampling step (S300) to which the background removal step (S330) is added will be described in detail with reference to FIG.

  FIG. 6 is an exemplary diagram illustrating a method for measuring a movement amount of a first learning target object using a frame image in which the background is removed in the secondary sampling step of FIG. 1 to which a background removal step is added. is there.

  Referring to FIG. 6, FIG. 6A illustrates the frame image 10 extracted in the first sampling step S200, and the frame image 10 includes a first learning target object 20 and a background 30.

  Referring to FIG. 6, FIG. 6B is a view illustrating that the background 30 is removed in the background removal step S330 of the frame image 10 of FIG. 6A. Only the first learning target object 20 is included.

  Referring to FIG. 6, FIG. 6C illustrates positions of one or more first learning target objects 21 detected on an n-th (n is a natural number of 2 or more) frame image and an n-th frame image. It is a figure which shows comparing with the position of one or more 1st learning target objects 22 detected on the n-1st frame image from which the background just before was removed.

  The computer can measure the movement amount of the first learning target object by comparing the first learning target objects 21 and 22.

  After measuring the amount of movement by comparing the positions of the first learning target objects 21 and 22, the computer determines a predetermined number or more of the first learning target objects 20 among the detected one or more first learning target objects 20. If the amount of movement is equal to or greater than a predetermined reference, the nth frame image is selected as a learning target frame image, and if not, the nth frame image is not selected as a learning target frame image.

  For example, the computer compares the movement amount of the first learning target object detected on the first frame image with the movement amount of the first learning target object detected on the second frame image, and the movement amount is predetermined. If it is equal to or more than the reference, the second frame image is selected as the learning target frame image. When the amount of movement does not correspond to the predetermined reference or more, the second frame image is not selected as the learning target frame image.

  Further, the computer is not limited to selecting or not selecting the second frame image as the learning target frame image, and again, the first learning target object detected on the second frame image and the first learning target object detected on the third frame image. The movement amount of one learning target object is compared, and when the movement amount is equal to or more than a predetermined reference, the third frame image is selected as the learning target frame image.

  The step of measuring the amount of movement of the first learning target object on the frame image and selecting or not selecting it as the learning target frame image is the n-th (n is a natural number of 2 or more) one or more detected ones. By comparing the positions of the one learning target object and the detected one or more first learning target objects on the (n-1) th frame image, the detected one or more first learning target objects on all the extracted frame images. This is repeated until one learning target object is compared and selection or non-selection as a learning target frame is completed.

  When there are one or more first learning target objects 20 and the computer has a plurality of first learning target objects 20 on one frame image 10, the first learning target objects 20 are compared with each other. The movement amount of the first learning target object 20 is measured.

  For example, when a plurality of first learning target objects 20 are included in one frame image 10, the computer measures the movement amount of all of the plurality of first learning target objects 20 to determine a predetermined number of first learning targets. When the amount of movement of the object 20 is equal to or larger than a predetermined reference, the corresponding frame image 10 is selected as the learning target frame image.

  In one embodiment, the computer measures the movement amount of all of the plurality of first learning target objects 20, and when the movement amount of all of the plurality of first learning target objects 20 is equal to or more than a predetermined reference, the corresponding frame image 10 Is selected as the learning target frame image.

  In another embodiment, the computer measures only the movement amount of a predetermined number of first learning target objects 20 among the plurality of first learning target objects 20, and measures the measured movement amount of the first learning target object 20. If the movement amount of the predetermined number of first learning target objects 20 is equal to or larger than the predetermined reference, the corresponding frame image 10 is selected as the learning target frame image.

  In yet another embodiment, the computer measures only the movement amount of a predetermined number of first learning target objects 20 among the plurality of first learning target objects 20, and measures all the measured first learning target objects 20. When the amount of movement is equal to or greater than the predetermined standard, the corresponding frame image 10 is selected as the learning target frame image.

  The learning target frame image selected by the movement amount measurement in FIG. 6C is selected from the extracted frame images 11, 12, 13, 14, and 15 as illustrated in FIG. 4D. The learning target frame images 12 and 14 are selected.

  FIG. 7 is a schematic flowchart of a learning target image extraction method for autonomous driving to which a congestion variable calculation step is added according to another embodiment of the present invention.

  Referring to FIG. 7, the congestion variable calculation step (S400) is performed after the original moving image reception step (S100).

  In operation S400, the congestion variable calculation unit 640 calculates the congestion variable of the area associated with the original road driving image. The congestion variable indicates the degree to which the population or vehicles are densely associated with a specific area. For example, if the crowded variable is relatively high, it can be analyzed that a relatively large amount of data needs to be learned in order to autonomously drive the area. The congestion variable calculation unit 640 calculates the congestion variable of the specific area using one or more of the demographic data or the vehicle statistical data for the specific area. The demographic data may include, but is not limited to, at least one of a population density, a population distribution, and a floating population corresponding to a position and time when the road driving image is captured. In addition, the vehicle statistical data may include at least one of the number of registered vehicles corresponding to the position and time at which the road driving image was captured, traffic statistics by time zone, and public transportation means distribution. Not limited. The congestion variable calculation unit 640 calculates the congestion variable of the area associated with the original road driving image using at least one of the demographic data or the vehicle statistical data of the area associated with the original road traveling image.

  The first sampling unit 620 further determines the number of extractions based on the congestion variable of the area related to the original road running video in addition to the metadata, and extracts the determined number of frame images from the original road running video.

  In an exemplary embodiment, the first sampling unit 620 may extract the frame images by a number equal to or larger than a predetermined reference when the congestion variable is equal to or larger than the predetermined reference. For example, assume that the congestion variable can have a value of 1-10. The first sampling unit 620 may extract 30 frame images per second in an area where the dense variable is 10 and is relatively high, and may extract 15 frame images per second in an area where the dense variable is 5 and is relatively low. .

  The first sampling unit 620 may receive the dense variable corresponding to the demographic data or the vehicle statistical data via the public data server, but is not limited thereto.

  FIG. 8 is a schematic flowchart of a learning target image extraction method for autonomous driving to which a learning data set generating step is added according to another embodiment of the present invention.

  Referring to FIG. 8, the learning data set generating step (S500) is performed after the secondary sampling step (S300).

  In step S500, the learning data set generation unit 650 processes one or more learning target frame images to generate a learning data set. The step S500 will be described in detail with reference to FIG.

  FIG. 9 is a schematic flowchart of the learning data set generation step of FIG.

  Referring to FIG. 9, the learning data set generating step (S500) includes a second learning target object detecting step (S510) and a second learning target object annotating step (S520).

  In step S510, the learning data set generation unit 650 detects one or more second learning target objects of one or more predetermined types on the learning target frame image.

  The second learning target object is an object included in the learning target frame image. The learning target frame image includes one or more second learning target objects and one or more types of first learning target objects. For example, the second learning target object includes a person, an automobile, a bicycle, a building, a telephone pole, a motorcycle, a tree, a flower, a puppy, a cat, a road, a traffic display board, a step for preventing overspeed, a road cone, a lane, and the like. However, the present invention is not limited to the above example, and includes all objects that can be distinguished as objects. Further, the type of the second learning target object includes the front surface, the rear surface, the right side surface, the left side surface, etc. of the object, but the type of each learning target object is not limited to the above example, and is divided into more subdivided categories than the above example. And can be classified as a completely different type from the above example.

  If it is determined in step S300 that the detection of the first learning target object is to select an image for learning artificial intelligence, that is, to select a learning target frame image, in step S510, 2 The detection of the learning target object is for selecting the target for the annotation process.

  Detecting one or more objects of one or more types in the detection of the second learning target object is to detect using an object detection algorithm, and the object detection algorithm may include an R-CNN model. Yes, but not limited to.

  In step S520, the learning data set generation unit 650 performs annotation processing on the detected one or more second learning target objects on the learning target frame image.

  The annotation process means a work of generating data for explaining the detected second learning target object on the learning target frame image. The annotation process includes performing labeling, coloring, or layering on the second learning target object. Displaying what the second learning target object is can be included as an annotation process. For example, the one or more second learning target objects detected on the learning target frame image can be displayed by dividing the area according to a form such as a box. The annotation process can utilize tools that are automatically processed.

  An explanation of what each second learning target object is can be created for a region that is divided and displayed as a second learning target object, and the labeling may be created by one word. It may be created in detail in sentences, not in one word.

  The annotation process creates a database of the information labeled, colored or layered by one or more workers, generates a module learned by a computer, and uses the learned module on the newly input learning target frame image. It is possible to perform labeling, coloring, or layering on the second learning target object by determining whether the second learning target object is similar to the second learning target object stored in the database.

  In another embodiment, the learning data set generation unit 650 determines the importance of one or more second learning target objects detected on the learning target frame image using video processing, and determines the importance to be predetermined. If it is equal to or more than the reference, the annotation information for the second learning target object can be generated. When there are one or more second learning target objects, the learning data set generation unit 650 can generate annotation information only for the second learning target objects having high importance. For example, when there are a car and a traffic signal on the learning target frame image, it is determined that the relevant car and the traffic signal have high importance, and annotation information can be generated only for the corresponding learning car and the traffic signal.

  FIG. 10 is a schematic flowchart of the learning data set generation step of FIG. 8 to which the validity verification step is added.

  Referring to FIG. 10, the validation step (S530) is performed after the learning target object annotation step.

  In step S530, the inspector verifies the validity of the training data set. The inspector terminal receives the learning data set from the learning target image extraction device 600 for autonomous traveling. Step S530 will be described in detail with reference to FIG.

  FIG. 11 is a schematic flowchart of the validity verification step of FIG.

  Referring to FIG. 11, the validity verification step (S530) includes a step of requesting the examination of the learning data set (S531) and a step of examining the learning data set (S532).

  In step S531, the learning target image extraction device 600 for autonomous driving requests the inspector terminal to perform an inspection.

  In step S532, the inspector receives the learning data set and performs the inspection via the inspector terminal.

  In one embodiment, when the inspection result of the learning data set, the reliability of the learning data set is equal to or higher than a predetermined reference, the inspector terminal determines the learning data set and the corresponding reliability as learning targets for autonomous driving. The image is transmitted to the image extracting device 600. Therefore, the learning target image extraction device 600 for autonomous traveling can confirm that the corresponding learning data set is a data set suitable for use with the artificial intelligence module.

  In one embodiment, when the inspection result of the learning data set and the reliability of the learning data set are less than a predetermined standard, the corresponding learning data set is returned. Then, the inspector terminal requests the learning target image extraction device 600 for autonomous traveling to re-work the corresponding learning data set. The learning data set reworked by the learning target image extracting apparatus 600 for autonomous driving is re-examined in steps S531 to S532.

  FIG. 12 is a schematic configuration diagram of a learning target image extraction apparatus for autonomous traveling according to another embodiment of the present invention.

  Referring to FIG. 12, the learning target image extracting device 600 for autonomous traveling includes an original road traveling video receiving unit 610, a first sampling unit 620, a second sampling unit 630, a congestion variable calculation unit 640, And a learning data set generation unit 650.

  The components of the learning target image extracting apparatus 600 for autonomous driving in FIG. 12 correspond to the components of the method described with reference to FIGS. 1 to 11. The original road driving image receiving unit 610 performs an original moving image receiving step (S100), the first sampling unit 620 performs a primary sampling step (S200), and the second sampling unit 630 performs a secondary sampling step (S300). After that, the dense variable calculation unit 640 can execute the dense variable calculation step (S400), and the learning data set generation unit 650 can execute the learning data set generation step (S500). In the following, a duplicate description will be omitted in describing the function or operation of each component of device 600.

  The original road traveling video receiving unit 610 receives the original road traveling video.

  The first sampling unit 620 determines the number of extractions based on the metadata of the original road driving video, and extracts the determined number of frame images from the original road driving video.

  The metadata of the original road traveling image includes one or more of GPS information of the vehicle related to the original road traveling image, moving speed information of the vehicle, and shooting time of the original road traveling image.

  The second sampling unit 630 may select one of the determined number of frame images based on whether or not each of the determined number of frame images includes the first learning target object and the movement amount of the first learning target object. The above learning target frame image is extracted.

  The congestion variable calculation unit 640 calculates the congestion variable of the area related to the original road traveling image.

  The learning data set generation unit 650 processes one or more learning target frame images to generate a learning data set.

  The steps of the method or algorithm described in connection with the embodiments of the present invention may be implemented directly in hardware, a software module executed by hardware, or a combination thereof. . The software module is a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Programmable Erasable ROM), a hard disk (ROM), a flash memory (FRAM). The present invention can also be present in a computer-readable recording medium in any form well known in the technical field to which the present invention belongs.

  Although the embodiments of the present invention have been described above with reference to the accompanying drawings, a person skilled in the art to which the present invention pertains does not change the technical idea or essential features of the present invention, and It should be understood that it can be implemented in a specific form of. Therefore, the embodiments described above are to be understood as being illustrative in all aspects and not restrictive.

600 Learning Target Image Extraction Device for Autonomous Driving 610 Original Road Running Video Reception Unit 620 First Sampling Unit 630 Second Sampling Unit 640 Congestion Variable Calculating Unit 650 Learning Data Set Generating Unit

Claims (10)

Original video receiving stage to receive the original road driving video,
A primary sampling step of determining an extraction number based on the metadata of the original road driving image and extracting the determined number of frame images from the original road driving image;
Based on whether or not each of the determined number of frame images includes a first learning target object, and based on the amount of movement of the first learning target object, one or more of the determined number of frame images are selected. A second sampling stage for extracting a frame image to be learned,
The metadata of the original road driving image includes one or more of GPS information of the vehicle that captured the original road driving image , moving speed information of the vehicle, and a shooting time of the original road driving image. Method of learning target image for learning. The primary sampling step is
Further comprising a crowded variable calculating step of calculating a crowded variable of an area associated with the original road driving image,
The method for extracting a learning target image for autonomous driving according to claim 1, wherein the extraction number is determined further based on the congestion variable of an area related to the original road driving image. The dense variable calculation step includes
The crowded variable of the area related to the original road driving image is calculated by using at least one of demographic data or vehicle statistical data of the area related to the original road driving image. A method for extracting a learning target image for the autonomous driving described above. The secondary sampling step includes
A first learning target object detecting step of detecting one or more first learning target objects of one or more predetermined types on each of the determined number of frame images using an object detection algorithm;
The detected position of the one or more first learning target objects on the n-th (n is a natural number of 2 or more) frame image and the detected position on the (n-1) -th frame image immediately before the n-th frame image. A first learning target object movement amount that compares the positions of one or more first learning target objects and measures the movement amount of each of the detected one or more first learning target objects on the n-th frame image Measurement stage,
The learning target image extraction method for autonomous traveling according to claim 1, further comprising: The secondary sampling step includes
The learning target image for autonomous driving according to claim 4, further comprising: a background removal step of removing a background excluding the first learning target object on each of the determined number of frame images. Extraction method.   The learning target image extraction for autonomous driving according to claim 1, further comprising: a learning data set generating step of processing the one or more learning target frame images to generate a learning data set. Method. The learning data set generating step includes
A second learning target object detecting step of detecting one or more second learning target objects of one or more predetermined types on the learning target frame image;
A learning target object annotation step of performing annotation processing on the one or more second learning target objects detected on the learning target frame image;
The learning target image extraction method for autonomous driving according to claim 6, further comprising: The learning data set generating step includes
The method of claim 7, further comprising an effectiveness verification step of verifying the effectiveness of the learning data set. An original road running video receiving unit that receives the original road running video,
A first sampling unit that determines an extraction number based on the metadata of the original road running video and extracts the determined number of frame images from the original road running video;
Based on whether or not each of the determined number of frame images includes a first learning target object, and based on the movement amount of the first learning target object, one or more of the determined number of frame images are selected. A second sampling unit for extracting a learning target frame image,
The metadata of the original road driving image includes one or more of GPS information of the vehicle that captured the original road driving image , moving speed information of the vehicle, and the shooting time of the original road driving image. Image extraction device for learning.   A computer program stored in a computer-readable recording medium for executing the learning target image extraction method for autonomous traveling according to any one of claims 1 to 8 in combination with a computer.

Images