RU2763215C2 - Methods and systems for forming training data for a neural network - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: group of inventions relates to a method and a computer apparatus for forming training data for re-training a neural network for detecting objects. In order to form training data, a first digital image is introduced into the neural network (object detecting Neural Network, ODNN), constituting a digital signal of the detected object at a first moment of time, and a second digital image at a second moment of time after the first moment of time, said images are compared in order to determine the degree of similarity. If the degree of similarity is below the predetermined threshold value, the first or the second image is used to generate a human-evaluated label indicating the real part of the first or second image, and the ODNN network is re-trained.

EFFECT: object detection network ODNN is re-trained.

32 cl, 7 dwg

Description

The field of technology to which the invention belongs

[1] The present technology relates to computer methods and machine learning systems, in particular, to methods and systems for generating training data for a neural network.

State of the art

[2] In the known solutions proposed and implemented computer navigation systems that facilitate navigation and/or control of the vehicle. Various systems of this kind are known, ranging from simple map-based location solutions using a computer system to assist the driver in navigating a route from origin to destination, to more complex solutions such as computerized and/or autonomous driving systems.

[3] Some of these systems are implemented in the well-known cruise control system. In this case, the computer system installed on the vehicle maintains the vehicle speed set by the user. Some cruise control systems implement an intelligent distance control system, allowing the user to set the distance to the vehicle in front (for example, select a value expressed in terms of the number of vehicles). The computer system then controls the speed of the vehicle, at least in part, depending on its approach to the vehicle ahead by a predetermined distance. Some of the cruise control systems are additionally equipped with collision avoidance systems that, when a vehicle (or other obstacle) is detected in front of a moving vehicle, will slow or stop it.

[4] Some of the most advanced systems provide fully autonomous vehicle movement without the direct involvement of the operator (i.e. driver) in control. Such autonomously driven vehicles contain computer systems capable of accelerating, decelerating, stopping, changing lanes, and self-parking the vehicle.

[5] One of the main technical problems encountered in the implementation of the above computer systems is the ability of the computer system to detect objects around the vehicle. For example, computer systems require the ability to detect another vehicle in front of a vehicle with a computer system installed that could pose a risk to that vehicle and require the computer system to take proactive action, such as braking or otherwise changing speed, stopping, or changing lanes.

[6] In particular, the problem of object detection is not only in binary detection (the presence or absence of an object), but also in determining its speed, as well as in the accuracy associated with such analysis and detection (especially in preventing false negatives when the system is not detects an object actually present in front of or near the vehicle).

[7] The importance of this problem is illustrated by the following hypothetical scenario. It can be assumed that a self-driving or partially autonomous vehicle is moving along the route. A child (or adult or pet, etc.) is running in front of the vehicle. The computer system controlling the vehicle is required to quickly detect the presence of an object and take corrective action to avoid a collision. Of course, the faster the computer system detects the presence of an object, the more time it has to determine corrective action and instruct the vehicle to take that corrective action.

[8] It should also be noted that the computer system that controls the vehicle must also accurately determine the boundaries of the object. Of course, the boundaries of objects, such as traffic lanes, must be precisely defined in order to determine the location of the vehicle in relation to the traffic lanes and, if necessary, take corrective actions. In other words, the task of object detection is not only the classification of objects (for example, a car, a person, a tree, etc.), but also the localization of these objects.

Disclosure of invention

[9] Developers of the present technology have found that prior art object detection solutions have disadvantages. In particular, known technologies may inconsistently localize objects when they are detected. For example, the location of an object in one image and the location of the same object in an almost identical image may be inconsistent.

[10] The developers of the present technology have found that significant differences in predicting the location of an object in almost identical images present a certain problem, and this may indicate incorrect training of the machine learning algorithm used for prediction. In the case where object detection is used to control the trajectory of a vehicle, the unequal localization of objects can be dangerous, since the vehicle can be controlled based on the displaced location of the detected object.

[11] Therefore, the developers of the present technology have developed methods and computer devices for retraining Object Detecting Neural Networks (ODNNs) so that they can localize objects in images in a more consistent manner, i.e. the ODNN network can be retrained so that for similar images of an object, close locations of this object in similar images are determined.

[12] In some embodiments of the present technology, developers provide the ability to generate, using a computer device, a data set for retraining an ODNN network based on a digital signal from a sensor, such as a camera, that captures images of the area around a moving vehicle on which this sensor is installed. In other words, the developers of this technology provide that the sensor can capture a digital signal while the vehicle is moving, and based on this digital signal, the computer device can form a data set for retraining the ODNN network, which allows more consistent localization of objects in images.

[13] In addition, the developers of the present technology provide that the computer device can determine the zones of inconsistent detection on the route of the vehicle. For example, the computing device may identify and track geographic locations where object localization from images acquired for those geographic locations was inconsistent.

[14] In other words, non-limiting examples of the present technology aim to generate a retraining dataset focused on samples for which the ODNN made an inconsistent prediction based on two images with substantially similar features (i.e., two images, adjacent to each other in time and space). Thus, non-limiting embodiments of the present technology allow a retraining dataset to be generated that focuses the ODNN on "hard to train" areas, i. on areas where the ODNN tends to mispredict.

[15] It should be noted that in some embodiments of the present technology, a computing device may use an ODNN for object detection. It should also be noted that this computing device can use the data provided by the ODNN network to control the vehicle. For example, the vehicle may be a semi-autonomous or fully autonomous vehicle controlled by a computing device at least in part based on data provided by the ODNN.

[16] In addition, it is contemplated that a computer device can control such a vehicle, at least in part, based on data provided by the ODNN, without human intervention. The computer device may control the vehicle based at least in part on data provided by the ODNN and not require human analysis of the data provided by the ODNN. The computer device may control the vehicle based at least in part on data provided by the ODNN and not require a human decision to be made based on the data provided by the ODNN. The computer device may control the vehicle based at least in part on data provided by the ODNN and not require human control of the vehicle. The computer device may steer the vehicle based at least in part on predictions made by the ODNN and not require a human decision based on those predictions to steer the vehicle.

[17] In accordance with the first aspect of the present technology, a method for generating a dataset for retraining an ODNN network is implemented. The ODNN is trained to detect objects in a digital signal captured by a moving vehicle by determining the part of the digital image corresponding to these objects. The method may be performed by a computer device. The method includes inputting a first digital image into the ODNN. The first digital image is the digital signal at the first time. The ODNN is capable of detecting an object in the first digital image by determining the first part of the first digital image corresponding to the object. The method includes inputting a second digital image into the ODNN. The second digital image is a digital signal at a second time after the first time. The ODNN is capable of detecting an object in the second digital image by determining the second part of the second digital image corresponding to this object. The method includes comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity of detection of this object. The degree of similarity of detection indicates the degree of similarity between the predictions made by the ODNN at the first time and at the second time when this object is found. When the degree of similarity of detection decreases below a predetermined threshold, the method includes using at least one of the first digital image and the second digital image to obtain a human-rated label indicative of a real part of at least one of the first digital image and the second digital image, occupied by this object on the corresponding one of the first digital image and the second digital image. The method includes retraining the ODNN based on at least one of the first digital image and the second digital image and a human-rated label.

[18] In some embodiments of the method, the computing device may be implemented as a processor. For example, the method may be performed by the processor of the electronic device. In another example, the method may be performed by a server processor.

[19] In some embodiments of the method, comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity of detection includes the use of analysis of intersection and union of sets (IOU, Intersection Over Union).

[20] In some embodiments of the method, applying the IOU analysis includes determining the intersection of the first part and the second part, and determining the union of the first part and the second part.

[21] In some embodiments of the method, it further includes receiving from the human appraiser an indication of the actual portion occupied by the object. The real part occupied by an object is determined by the human appraiser based on the actual state of that object.

[22] In some embodiments, the method further includes receiving from the human appraiser an indication of the actual portion occupied by the object. The real part occupied by an object is determined by a human appraiser based on an artificial change in the state of this object.

[23] In some embodiments of the method, retraining of the ODNN is performed to detect objects in accordance with the actual states of the objects.

[24] In some embodiments of the method, ODNN retraining is performed to detect objects in accordance with artificially changed object states.

[25] In some embodiments of the method, ODNN retraining is performed based on the first digital image, the second digital image, and the corresponding human-rated labels.

[26] In some embodiments of the method, it further includes inputting a third digital image into the ODNN. The third digital image is the digital signal at the third time point. The ODNN is capable of detecting an object in the third digital image by determining the third part of the third digital image corresponding to this object. The method further includes inputting a fourth digital image to the ODNN. The fourth digital image is a digital signal at the fourth time point after the third time point. The ODNN is capable of detecting an object in the fourth digital image by determining the fourth part of the fourth digital image corresponding to this object. The method further includes comparing the third part of the third digital image with the fourth part of the fourth digital image to determine the degree of similarity of detection of this object. The degree of similarity of detection indicates the degree of similarity between the predictions made by the ODNN at the third time and at the fourth time when this object was found. When the degree of similarity of detection decreases below a predetermined threshold value, the method further includes using at least one of the third digital image and the fourth digital image to obtain a human-rated label indicative of a real part of at least one of the third digital image and the fourth digital image. occupied by this object on the corresponding one of the third digital image and the fourth digital image.

[27] In some method embodiments, at least one of a first digital image and a second digital image with a corresponding human-rated label and at least one of a third digital image and a fourth digital image with a corresponding human-rated label form, at least in part, data set for retraining the ODNN network.

[28] In some embodiments of the method, the ODNN retraining dataset is used to retrain the ODNN on digital image features in which the ODNN's object detection is inconsistent.

[29] In some embodiments of the method, the ODNN is trained to detect objects based on the training digital image and a human-rated label for the objects in the training digital image, wherein the ODNN predicts object classes and object locations in the training digital image.

[30] In some embodiments of the method, the area around the moving vehicle at at least one of the first time, the second time, the third time, the fourth time is defined as the inconsistent detection area.

[31] In some embodiments of the method, ODNN retraining is performed to reduce the likelihood of inconsistent detection of an object by the ODNN when the vehicle is in an inconsistent detection region.

[32] In some embodiments of the method, ODNN retraining is performed to reduce the likelihood of inconsistent object detection by the ODNN based on digital image features captured in the inconsistent detection area.

[33] In some embodiments of the method, ODNN retraining is performed to reduce the likelihood of inconsistent detection of an object by the ODNN based on digital image features captured in areas like the inconsistent detection area.

[34] According to a second aspect of the present technology, a computer device is implemented for generating training data for retraining an ODNN. The ODNN is trained to detect objects in a digital signal captured by a moving vehicle by determining the part of the digital image corresponding to these objects. The computing device is capable of inputting a first digital image into the ODNN. The first digital image is the digital signal at the first time. The ODNN is capable of detecting an object in the first digital image by determining the first part of the first digital image corresponding to the object. The computing device is capable of inserting a second digital image into the ODNN. The second digital image is a digital signal at a second time after the first time. The ODNN is capable of detecting an object in the second digital image by determining the second part of the second digital image corresponding to this object. The computer device is capable of comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity in detecting that object. The degree of similarity of detection indicates the degree of similarity between the predictions made by the ODNN at the first time and at the second time when this object is found. When the degree of similarity of detection decreases below a predetermined threshold value, the computing device is configured to use at least one of the first digital image and the second digital image to obtain a human-rated label indicating the real part of at least one of the first digital image and the second digital image occupied by this object in at least one of the corresponding first digital image and the second digital image. The computing device is capable of retraining the ODNN based on at least one of the first digital image and the second digital image and/or a human-rated label.

[35] In some embodiments of the computing device, comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity of detection suggests the ability of the computing device to apply IOU analysis.

[36] In some embodiments of the computing device, applying the IOU analysis includes determining the intersection of the first part and the second part, and determining the union of the first part and the second part.

[37] In some embodiments of the computing device, it is further capable of receiving from the human evaluator an indication of the actual portion occupied by the object. The real part occupied by an object is determined by the human appraiser based on the actual state of that object.

[38] In some embodiments of the computing device, it is further capable of receiving from the human evaluator an indication of the actual portion occupied by the object. The real part occupied by an object is determined by a human appraiser based on the artificially altered state of that object.

[39] In some embodiments of a computing device, ODNN retraining is performed to detect objects according to the actual states of those objects.

[40] In some embodiments of a computing device, ODNN retraining is performed to detect objects in accordance with artificially altered object states.

[41] In some embodiments of the computing device, ODNN retraining is performed based on the first digital image, the second digital image, and the corresponding human-rated labels.

[42] In some embodiments of the computing device, the computing device is further capable of inserting a third digital image into the ODNN. The third digital image is the digital signal at the third time point. The ODNN is capable of detecting an object in the third digital image by determining the third part of the third digital image corresponding to this object. The computing device is capable of inserting a fourth digital image into the ODNN. The fourth digital image is a digital signal at the fourth time point after the third time point. The ODNN is capable of detecting an object in the fourth digital image by determining the fourth part of the fourth digital image corresponding to this object. The computer device is additionally configured to compare the third part of the third digital image with the fourth part of the fourth digital image to determine the degree of similarity of the detection of this object. The degree of similarity of detection indicates the degree of similarity between the predictions made by the ODNN at the third time and at the fourth time when this object was found. When the degree of similarity of detection decreases below a predetermined threshold value, the computing device is further able to use at least one of the third digital image and the fourth digital image to obtain a human-rated label indicating a real part of at least one of the third digital image and the fourth digital image, occupied by this object on the corresponding one of the third digital image and the fourth digital image.

[43] In some embodiments of a computing device, at least one of the first digital image and/or the second digital image with a corresponding human-rated label and at least one of the third digital image and the fourth digital image with a corresponding human-rated label form at least in part, a dataset for retraining an ODNN network.

[44] In some embodiments of a computing device, the ODNN retraining dataset is used to retrain the ODNN for digital image features in which the ODNN's object detection is inconsistent.

[45] In some embodiments of a computing device, an ODNN is trained to detect objects based on a training digital image and a human-rated label for objects in that training digital image, wherein the ODNN predicts object classes and object locations in that training digital image.

[46] In some embodiments of the computing device, the area around the moving vehicle at one of the first time, the second time, the third time, the fourth time is defined as a mismatched detection area.

[47] In some embodiments of a computing device, ODNN retraining is performed to reduce the likelihood of inconsistent detection of an object by the ODNN when a vehicle is in an inconsistent detection region.

[48] In some embodiments of a computing device, ODNN retraining is performed to reduce the likelihood of inconsistent object detection by the ODNN based on digital image features captured in the inconsistent detection area.

[49] In some embodiments of a computing device, ODNN retraining is performed to reduce the likelihood of inconsistent detection of an object by the ODNN based on digital image features captured in regions like the inconsistent detection region.

[50] In the context of this description, the term "server" means a computer program executed by appropriate hardware and capable of receiving requests (for example, from client devices) via a network and fulfill these requests or initiate their execution. The hardware may be implemented as a single physical computer or a single computer system, which is not essential to the present technology. In the present context, the expression "server" does not mean that every task (e.g. received command or request) or some particular task is received, executed or started by the same server (i.e. the same software and/or hardware ). This expression means that any number of software or hardware can receive, send, execute or start the execution of any task or request, or the results of any tasks or requests. All of these software and hardware may be a single server or multiple servers, both of which are meant by the expression "at least one server".

[51] In the context of this description, the term "electronic device" means any computer hardware capable of executing programs suitable for solving the task. In the context of this description, the term "electronic device" implies that the device can function as a server for other electronic devices and client devices, however, this is not necessary for the present technology. Thus, some (non-limiting) examples of electronic devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be understood that in the present context, the fact that a device functions as an electronic device does not mean that it cannot function as a server for other electronic devices. The use of the term "electronic device" does not preclude the use of multiple client devices to receive, send, perform or initiate any task or request or the results of any tasks or requests or steps of any method described herein.

[52] In the context of this description, the term "client device" means any computer hardware capable of executing programs suitable for solving the task. In the context of this description, the term "client device" is generally associated with the user of the client device. Thus, some (non-limiting) examples of client devices include personal computers (desktops, laptops, netbooks, and the like), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be noted that in this context, a device functioning as a client device may also function as a server for other client devices. The use of the term "client device" does not preclude the use of multiple client devices to receive, send, perform or initiate any task or request or the results of any tasks or requests or steps of any method described herein.

[53] In the context of this description, the expression "information" includes information of any kind or kind that can be stored in a database. Thus, information includes audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages etc.), but not limited to documents, spreadsheets, etc.

[54] In the context of this description, the expression "software component" includes the designation of software (suitable for certain hardware) necessary and sufficient to perform a particular function or several functions.

[55] In the context of this description, the expression "computer storage medium" (also called "information storage medium") means media of any kind and kind, including random access memory devices (RAM), read-only memory devices (ROM), disks (CD-ROM, DVD , floppy disks, hard drives, etc.), USB drives, solid state drives, tape drives, etc. A plurality of elements may be combined to form a computer storage medium, including two or more storage medium elements of the same kind and/or two or more storage medium elements of different kinds.

[56] In the context of this description, the term "database" means any structured set of data, regardless of its specific structure, database management software or computer hardware for storing this data, using them or providing them with use in another way. The database may reside on the same hardware as the process for storing or using the information stored in the database, or the database may reside on separate hardware such as a specialized server or multiple servers.

[57] In the context of this description, the numerals "first", "second", "third", etc. are used only to indicate the difference between the nouns to which they refer, and not to describe any specific relationship between these nouns. For example, it should be clear that the use of the terms "first server" and "third server" does not imply any particular order, kind, chronology, hierarchy or classification of, in this case, servers, and that their use (by itself) does not imply a "second server" in every situation. In addition, as occurs herein in another context, reference to a "first" element and a "second" element does not exclude that the two elements may in fact be the same element. Thus, for example, in some cases the "first" server and the "second" server may be the same piece of software and/or hardware, and in other cases they may be different pieces of software and/or hardware.

[58] Each embodiment of the present technology relates to at least one of the above objectives and/or aspects, but not necessarily all of them. It should be understood that some aspects of the present technology, associated with an attempt to achieve the above goal, may not meet this goal and/or may meet other goals not explicitly mentioned here.

[59] Additional and/or alternative features, aspects, and advantages of embodiments of the present technology are contained in the following description, in the accompanying drawings, and in the claims.

Brief description of the drawings

[60] These and other features, aspects and advantages of the present technology are explained in the following description, in the appended claims and in the following drawings.

[61] FIG. 1 is a diagram of an exemplary computer system for implementing some embodiments of systems and/or methods in accordance with the present technology.

[62] FIG. 2 depicts a networked computing environment suitable for use with some embodiments of the present technology.

[63] FIG. 3 shows the first digital image captured by the network computing environment sensor of FIG. 2 in accordance with non-limiting embodiments of the present technology.

[64] FIG. 4 shows the first digital image from FIG. 3 and first use phase detection data outputted by an Object Detection Neural Network (ODNN) implemented in the network computing environment of FIG. 2 in accordance with non-limiting embodiments of the present technology.

[65] FIG. 5 shows the first digital image captured by the sensor of FIG. 2 in accordance with non-limiting embodiments of the present technology.

[66] FIG. 6 shows the second digital image from FIG. 5 and second use stage discovery data issued by the ODNN in accordance with non-limiting embodiments of the present technology.

[67] FIG. 7 is a flow diagram of a method performed by a computing device in accordance with non-limiting embodiments of the present technology.

Implementation of the invention

[68] The examples and conventions presented herein are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specific examples and terms. It is obvious that specialists in the art are able to develop various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its essence and scope.

[69] In addition, to facilitate a better understanding, the following description may contain simplified implementations of the present technology. Those skilled in the art will appreciate that other embodiments of the present technology may be significantly more complex.

[70] In some cases, useful examples of modifications to the present technology are provided. They contribute to understanding, but also do not define the scope or boundaries of the present technology. The presented list of modifications is not exhaustive and the person skilled in the art can develop other modifications within the scope of this technology. In addition, if modifications are not described in some cases, this does not mean that they are impossible and / or that the description contains the only possible implementation of one or another element of the present technology.

[71] Moreover, the description of principles, aspects, and embodiments of the present technology, as well as their specific examples, is intended to cover their structural and functional equivalents, whether they are currently known or will be developed in the future. For example, it should be apparent to those skilled in the art that any structural diagrams described correspond to conceptual representations of illustrative circuit diagrams embodying the principles of the present technology. It should also be obvious that any flowcharts, process diagrams, state transition diagrams, pseudocodes, etc. correspond to various processes that may be represented on a computer-readable physical storage medium and may be performed by a computer or processor, whether such a computer or processor is explicitly shown or not.

[72] The functions of the various elements shown in the drawings, including any functional block designated as "processor", can be implemented using specialized hardware, as well as hardware capable of executing the corresponding software. If a processor is used, these functions may be performed by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. In addition, explicit use of the term "processor" or "controller" should not be construed as referring solely to the hardware capable of executing the software and may refer to, but is not limited to, digital signal processor (DSP) hardware, network processor, ASIC (ASIC), Field Programmable Gate Array (FPGA), Software ROM, RAM, and Non-Volatile Memory. Other general purpose and/or custom hardware may also be contemplated.

[73] Software modules, or simply modules that are intended to be implemented in software, may be represented here as any combination of flowchart elements or other elements that indicate the steps of a process and/or contain a textual description. Such modules may be implemented in hardware, as shown or implied.

[74] In the following, in light of the principles set forth above, some non-limiting examples are provided to illustrate various embodiments of aspects of the present technology.

computer system

[75] FIG. 1 depicts a computer system 100 suitable for use with some embodiments of the present technology and comprising various hardware elements including one or more single or multi-core processors collectively represented by processor 110, a solid state drive 120, and memory 130, which may be RAM or memory of any other kind.

[76] Communication between elements of computer system 100 may be via one or more internal and/or external buses (not shown) (such as PCI bus, Universal Serial Bus (USB), IEEE 1394 FireWire bus, SCSI bus, Serial- ATA, etc.) to which various hardware elements are connected electronically. In accordance with embodiments of the present technology, the solid state drive 120 stores program instructions suitable for loading into memory 130 and execution by processor 110 to determine the presence of an object. For example, software instructions may be part of a vehicle control application executed by processor 110. It should be noted that computer system 100 may include additional and/or optional elements (not shown), such as network communication modules, location modules, and etc.

networked computing environment

[77] FIG. 2 depicts a networked computing environment 200 suitable for use with some embodiments of systems and/or methods in accordance with the present technology. The networked computing environment 200 includes an electronic device 210 associated with a vehicle 220 and/or a user (not shown) who can operate the vehicle 220, a server 235 connected to the electronic device 210 via a communications network 240 (e.g., via the Internet and the like, as described in more detail below).

[78] The networked computing environment 200 may also include a GPS satellite (not shown) that transmits a GPS signal to and/or receives the signal from the electronic device 210. It should be understood that the present technology is not limited to the GPS system and may use a location technology other than the GPS system. It should be noted that the GPS satellite may not be available at all.

[79] The vehicle 220 with which the electronic device 210 is associated may be any recreational or transportation vehicle, such as a car for personal or commercial use, a truck, a motorcycle, and so on. Although depicted as a ground vehicle, vehicle 220 may be different in each embodiment of the present technology. For example, vehicle 220 may be a floating vehicle, such as a boat, or an aircraft, such as a flying drone.

[80] Vehicle 220 may be driven by a user or be a self-driving vehicle. It should be noted that no restrictions are imposed on specific parameters of the vehicle 200, such as vehicle manufacturer, vehicle model, vehicle year, vehicle weight, vehicle dimensions, vehicle weight distribution, vehicle surface area, vehicle height, transmission type (e.g., two or four wheel drive), tire type, brake system, fuel system, mileage, vehicle identification number, engine displacement, etc.

[81] The implementation of the electronic device 210 is also not subject to any restrictions. For example, the electronic device 210 may be implemented as a vehicle engine control unit, a vehicle central processing unit, a vehicle navigation device (eg, TomTom™, Garmin™), a tablet, a personal computer built into the vehicle 220, etc. . It should be noted that electronic device 210 may or may not be permanently associated with vehicle 220. Additionally or alternatively, electronic device 210 may be implemented in a wireless communication device such as a mobile phone (eg, smartphone or cordless phone). In some embodiments, the electronic device 210 includes a display 270.

[82] Electronic device 210 may include some or all of the elements of computer system 100 shown in FIG. 1. In some embodiments, the electronic device 210 is an on-board computer device and includes a processor 110, a solid state drive 120, and a memory 130. In other words, the electronic device 210 includes hardware and/or application software and/or firmware, or combination for data processing, as described in more detail below.

Sensor

[83] According to non-limiting embodiments of the present technology, the electronic device 210 further comprises or has access to a sensor 230 capable of capturing one or more digital images of at least a portion of the surrounding area 250 around the vehicle 220. The sensor 230 is coupled to the processor 110 to transfer the captured information to the processor 110 for processing, as described in more detail below.

[84] In a non-limiting example, sensor 230 includes a camera. There are no special restrictions on the implementation of the camera. For example, in some non-limiting embodiments of the present technology, the camera may be implemented as a black and white camera with sufficient resolution to detect objects at a predetermined distance of up to approximately 30 m (however, without departing from the scope of the present invention, cameras with a different resolution and a different range).

[85] The sensor 230 may be mounted internally at the top of the windshield of the vehicle 220, but other locations are also possible within the scope of the present invention, including the rear window, side windows, front hood, roof, front grille, or front bumper of the vehicle. means 220. In some non-limiting embodiments of the present technology, the sensor 230 may be located in a special housing (not shown) mounted on top of the vehicle 220.

[86] In some non-limiting embodiments of the present technology, sensor 230 may be implemented as a plurality of cameras. For example, the plurality of cameras may include enough cameras to capture a panoramic digital image of the surrounding area 250.

[87] In some embodiments of the present technology, the camera (one or more cameras in which sensor 230 is implemented) is capable of capturing a predetermined portion of the surrounding area 250 around the vehicle 220. In some embodiments of the present technology, the camera is capable of capturing a digital image (or sequence digital images) corresponding to a sector of the surrounding area 250 around the vehicle 220 at approximately 90 degrees along the path of the vehicle 220.

[88] In other embodiments of the present technology, the camera is capable of capturing a digital image (or sequence of digital images) corresponding to a sector of the surrounding area 250 around the vehicle 220 approximately 180 degrees along the vehicle 220's path.

[89] In other embodiments of the present technology, the camera is capable of capturing a digital image (or sequence of digital images) corresponding to a sector of the surrounding area 250 around the vehicle 220 approximately 360 degrees along the path of the vehicle 220 (in other words, the entire surrounding area around the vehicle 220).

[90] For example, in FIG. 3 shows the first digital image 300 captured by the sensor 230. The first digital image 300 represents a portion of the surrounding area 250 around the vehicle 220 at the first time. As seen in the first digital image 300, the vehicle 220 is moving on the road and another vehicle (e.g., a truck) is in the surrounding area 250 around the vehicle 220.

[91] In another example, in FIG. 5 shows a second digital image 500 captured by a sensor 230. The second digital image 500 represents a portion of the surrounding area 250 around the vehicle 220 at a second time after the first time. As seen in the second digital image 500, the vehicle 220 is also moving on the road and another vehicle is also in the surrounding area 250 around the vehicle 220.

[92] It should be noted that the first digital image 300 and the second digital image 500 may be part of a digital signal captured by the sensor 230. In other words, the sensor 230 may capture the digital signal (eg, as a sequence of digital images). Thus, the first digital image 300 may be the digital signal of the sensor 230 at the first time, and the second digital image 500 may be the digital signal of the sensor 230 at the second time.

[93] In a specific non-limiting example, sensor 230 is implemented as a camera available from FLIR Integrated Imaging Solutions Inc., 12051 Riverside Way, Richmond, BC, V6W 1K7, Canada. Obviously, the sensor 230 can be implemented in any other suitable equipment.

[94] It should be noted that in some embodiments of the present technology, additional sensors may be implemented in addition to sensor 230. For example, radar systems may be installed on vehicle 220 and coupled to processor 110. In another example, the vehicle 220 can be installed and connected to the processor 110 lidar systems - laser detection and ranging systems (LIDAR, LIght Detection And Ranging). Thus, the vehicle 220, for simplicity, is represented in FIG. 2 with only sensor 230, however, in other embodiments of the present technology, within its spirit and scope, vehicle 220 may be implemented with additional sensors in addition to sensor 230.

[95] In some embodiments of the present technology, sensor 230 may be calibrated. Such calibration may be performed during manufacture and/or customization of vehicle 220, or at any convenient time thereafter. In other words, the calibration may be performed during the upgrade of the vehicle 220 with the sensor 230 in accordance with the non-limiting embodiments of the present technology provided herein. Alternatively, calibration may be performed while vehicle 220 is equipped with sensor 230 in accordance with the non-limiting embodiments of the present technology provided herein.

Communication network

[96] In some embodiments of the present technology, communication network 245 is the Internet. In alternative non-limiting embodiments of the invention, the communications network may be implemented as any suitable local area network (LAN, Local Area Network), wide area network (WAN, Wide Area Network), private communications network, and the like. Obviously, the embodiments of the communication network are given for illustrative purposes only. The implementation of a communication link (not specifically indicated) between electronic device 210 and communication network 245 depends, among other things, on the implementation of electronic device 210. By way of non-limiting example only, in those embodiments of the present technology where electronic device 210 is implemented in in the form of a wireless communication device such as a smartphone or a navigation device, the communication link may be implemented as a wireless communication link. Examples of wireless links include a 3G network link, a 4G network link, and the like. Communication network 245 may also use a wireless connection to server 235.

Server

[97] In some embodiments of the present technology, the server 235 is implemented as a conventional computer server and may contain some or all of the elements of the computer system 100 shown in FIG. 1. In one non-limiting example, server 215 is implemented as a Dell™ PowerEdge™ server running the Microsoft™ Windows Server™ operating system, but it may also be implemented using any other suitable hardware, application software, and/ or firmware, or a combination thereof. In the present non-limiting embodiments of the present technology, the server is a single server. In other non-limiting embodiments of the present technology (not shown), the functions of the server 235 may be distributed among multiple servers.

[98] In some non-limiting embodiments of the present technology, processor 110 of device 210 may be connected to server 235 to receive one or more updates. The updates may be software updates, map updates, route updates, weather data updates, etc. In some embodiments of the present technology, processor 110 may also send some operational data to server 235, such as routes traveled, traffic data, performance data, and so on. Some or all of the data transmitted between vehicle 220 and server 235 may be encrypted and/or anonymized.

CPU

[99] The processor 110 is connected to the sensor 230 to receive image data from it. Processor 110 has access to the ODNN. In general, the ODNN is a Neural Network (NN) capable of performing object detection based on image data received from sensor 230 (and possibly other image data sources). This means that processor 110 is capable of receiving image data representing a digital image from sensor 230 and can perform (a) classification of objects in the digital image and (b) localization of objects in the digital image using the ODNN.

[100] The ODNN can classify objects in a digital image into one or more object classes. Object classes can include "vehicle", "car", "truck", "person", "animal", "tree", "building", "road", "lane", "traffic sign" , "wall", "traffic light", etc., but not limited to this.

[101] In one embodiment of the invention, the ODNN can perform a classification of objects into the same class. This means that the ODNN can determine whether an object belongs to a certain object class. For example, the ODNN can determine whether this object belongs to the class "vehicle".

[102] In other embodiments of the invention, the ODNN can perform a classification of objects into multiple classes. This means that the ODNN can determine that an object belongs to one of many object classes. For example, the ODNN may determine whether an object belongs to one of the following object classes: "vehicle", "person", or "lane".

[103] The ODNN can perform localization of objects in a digital image by determining the portion of the digital image corresponding to (or occupied by) that object. For example, an ODNN may define the boundaries of an object in a digital image (and/or the boundaries of a portion of the digital image containing the object). In another example, the ODNN may determine the zone in which an object is located in a given digital image.

[104] The following describes how the ODNN can perform object discovery, i. classification and localization of objects in digital images.

Neural network

[105] As stated above, the ODNN is a neural network (NN). In the general case, the NN network consists of a group of interconnected artificial "neurons" that process information using a connection approach. NNs are used to model complex relationships between inputs and outputs (without actual knowledge of those relationships) or to look for patterns in data. The NNs are first adapted in the training phase when they are provided with a known set of inputs and information in order to adapt the NNs to generate the correct output (for the situation being attempted to simulate). During the training phase, the NN network adapts to the situation being studied and changes its structure so that it can provide adequate predictive output for the input in the new situation (based on what has been learned). Thus, instead of trying to determine complex statistical patterns or mathematical algorithms for a given situation, the NN tries to provide an "intuitive" answer based on the "perception" of the situation. Thus, the NN network is a kind of trained black box that can be used in situations where the contents of the box do not matter, but only that this box provides adequate answers to the input data.

[106] NNs are widely used in many of these situations where it is only important to derive output from input and less important or not at all important to know how that output is obtained. For example, NNs are widely used to optimize the distribution of web traffic between servers and data processing, including filtering, clustering, signal separation, compression, vectorization, speech recognition, and the like.

[107] NN networks can be divided into several classes, which may have different topology or architecture, different properties, and can be used in different applications. One class of NN networks includes Convolutional Neural Networks (CNNs). CNNs are multi-layer NNs that have an input layer, an output layer, and a set of hidden layers, which may (depending on their specific implementation) include, for example, convolutional layers, pooling layers, fully connected layers, and normalizing layers. layers. After deep learning, CNNs can be used for visual image analysis and other computer vision applications.

[108] It is contemplated that, in some embodiments of the present technology, the ODNN used by processor 110 for object detection purposes may be a CNN within the spirit and scope of the present technology.

[109] In general, the implementation of the ODNN network using the processor 110 can be broadly divided into two stages - the training stage and the use stage.

[110] First, the ODNN is trained in the training phase. During the training phase of the ODNN network, a large number of training iterations can be performed. In general, during the training iteration, the ODNN is fed with (a) image data representing the training digital image, and (b) human-rated label data (e.g., training human-rated label) for objects in the training digital image (e.g., data human evaluation of object detection). In particular, the human-rated label data may contain indications of the classes of objects identified by the human evaluators, as well as indications of the location of the objects identified by the human evaluators. Therefore, the ODNN network in a sense "learns" to identify objects, determine their classes and location on the training digital image based on the image data representing the training digital image. It is contemplated that the ODNN can be implemented and trained by at least one of the servers 235 and/or processors 110 of the electronic device 210.

[111] Then, in the use phase, when the ODNN knows what data to expect as input (for example, image data corresponding to digital images) and what data to output as output (for example, object detection data - object classes and the location of objects in digital images), the ODNN is started with the data of the use phase. In general, at the use stage, image data corresponding to the digital image of the use stage is entered into the ODNN, and in response, object detection data of the use stage is returned, determining (a) the presence of objects in the digital image of the use stage, (b) the classes of these objects, and ( c) the location of these objects on the digital image of the stage of use.

[112] The following describes in detail, shown in FIG. 3 and FIG. 4 is a first non-limiting example of an ODNN usage step. It should be recalled that the first digital image 300 corresponds to the digital signal of the sensor 230 at the first time. This means that the first digital image 300 may be a digital image (relating to a first point in time) in a sequence of digital images captured by sensor 230 and transmitted to processor 110 and/or server 235.

[113] The processor 110 may input the image data corresponding to the first digital image 300 into the ODNN. In other words, the processor 110 may use the image data corresponding to the first digital image 300 as the use stage data for the ODNN. In response, the ODNN is able to provide the first use stage discovery data 490 shown in FIG. 4. In this case, the first use phase discovery data 490 indicates that the ODNN has discovered the first object 400 and the second object 450.

[114] For example, the ODNN may provide first usage phase detection data 490 indicating that (a) the first object 400 is present in the first digital image 300, (b) the first object 400 belongs to the lane object class, and ( c) the first object 400 is located on the first digital image 300 within the first boundaries 402.

[115] It can be said that the ODNN is able to detect the first object 400 in the first digital image 300 by determining the first part 403 (for example, a set of pixels) of the first digital image 300 corresponding to the first object 400. For example, the first part 403 is a part of the first digital image 300 bounded by first borders 402.

[116] In addition, the ODNN is capable of providing first use phase detection data 490 indicating that (a) the second object 450 is present in the first digital image 300, (b) the second object 450 belongs to the lane object class, and (c) the second object 450 is located on the first digital image 300 within the second boundaries 452.

[117] It can be said that the ODNN is able to detect the second object 450 in the first digital image 300 by determining another part 453 (for example, a different set of pixels) of the first digital image 300 corresponding to the second object 450. For example, the other part 453 is part of the first digital image 300 bounded by second borders 452.

[118] In other words, it can be said that the ODNN detects the presence of two lanes on the road on which the vehicle 220 is traveling, where the rightmost lane of the road is the first object 400, and the leftmost lane of the road is the second object 450. In In some embodiments, the ODNN may also detect additional objects in the first digital image 300, such as an object corresponding to another vehicle (in particular, a truck). However, for simplicity, it can be assumed that the ODNN detects the first object 400 and the second object 450 in the first digital image 300.

[119] The following describes in detail, shown in FIG. 5 and FIG. 6 is a second non-limiting example of an ODNN usage step. It should be recalled that the second digital image 500 corresponds to the digital signal of the sensor 230 at the second time. The second digital image 500 may be a digital image (related to the second point in time) in the sequence of digital images captured by the sensor 230 and transmitted to the processor 110 and/or the server 235.

[120] In this case, the second time occurs after the first time associated with the first digital image 300. For example, the time interval between the first time and the second time may be on the order of a few milliseconds. In another example, the digital signal of the sensor 230 may be in video format, where the two frames of the digital signal are the respective frames of the first digital image 300 and the second digital image 500. In some embodiments, the first digital image 300 and the second digital image 500 may be consecutive digital images. images of the digital signal of the sensor 230, but this is not required for each implementation of the present technology.

[121] Processor 110 and/or server 235 may input image data corresponding to second digital image 500 into the ODNN. In other words, processor 110 and/or server 235 may use image data corresponding to first digital image 500 as use step data. for the ODNN network. In response, the ODNN may provide second usage stage discovery data 690 shown in FIG. 6. In this case, the second use-phase discovery data 690 indicates that the ODNN has detected a changed first object 400' instead of the first object 400 and the second object 450.

[122] For example, the ODNN may provide second usage phase detection data 690 indicating that (a) the changed first object 400' is present in the second digital image 500, (b) the changed first object 400' belongs to the "stripe" object class. movement" and (c) the changed first object 400' is located on the second digital image 500 within the third boundaries 502.

[123] It can be said that the ODNN is able to detect the changed first object 400' in the second digital image 500 by determining the second part 503 (eg, a set of pixels) of the second digital image 500 corresponding to the changed first object 400'. For example, the second part 503 is the part of the second digital image 500 delimited by the third borders 502.

[124] In other words, it can be said that the ODNN detects, based on the second digital image 500, the presence of one lane on the road, where this lane on the road is the modified first object 400', as opposed to detecting the presence of two lanes on the road based on the first digital image 300.

[125] It should be noted that both the modified first object 400' and the first object 400 detected by the ODNN correspond to the same real object (e.g., a particular lane), however, the modified first object 400' and the first object 400 were localized differently by the ODNN when they were discovered, ie. ODNN has identified different parts (eg, locations) of the corresponding digital images corresponding to the same real object.

[126] It should be noted that in these non-limiting examples of the ODNN step, it does not matter whether there are actually one or two lanes on the road on which the vehicle 220 is traveling. Regardless of whether there is one or two lanes on the road on which the vehicle 220 is traveling, the developers of the present technology have come to the conclusion that the detection by the ODNN network of this object in two digital images (which were captured with a small time difference) as having a significant different parts (eg locations) in the respective digital images is a problem. The reason for this, according to the developers of this technology, is that the ODNN network, having received almost the same images with almost the same features, nevertheless gives significantly different predictions.

[127] In other words, it does not matter which result of the detection performed by the ODNN is more accurate - on the first digital image 300 or on the second digital image 500. Regardless of which result of the detection performed by the ODNN is more accurate - on the first digital image 300 or on second digital image 500, the developers of the present technology concluded that significant differences in the detection results performed by the ODNN network on two images captured with a small difference in time indicate a problem.

[128] It should be recalled that the developers of the present technology have developed methods and systems for retraining the ODNN so that the ODNN detects objects in a more "consistent" manner, regardless of the actual conditions of the surrounding area 250 around the vehicle 220 (see Fig. 2).

[129] For example, it can be assumed that in reality there is only one lane on the road on which the vehicle 220 is traveling. It can also be assumed that this single lane is so wide that it allows two vehicles to drive side by side on it (even if this is not allowed by the rules). It may be useful in some implementations of the method to train the ODNN to detect one wide lane on the road in order to predict/detect actual road conditions during the use phase.

[130] However, it is also possible that in other implementations it is useful to train the ODNN to detect two lanes on the road, which does not correspond to real road conditions, but rather "artificially modified" road conditions. Detection of artificially changed road conditions (instead of real road conditions) can be useful to mitigate the consequences of risk arising, for example, from overtaking maneuvers on the road. Thus, the detection by the ODNN of two traffic lanes on the road (which are not real road conditions, but artificially modified road conditions) allows the processor 110 to control the vehicle 220 differently, including in terms of maneuvering, "waiting" for a possible overtaking by a second vehicle, even if there is only one lane on the road.

[131] Therefore, whether it is desirable to detect one or two lanes on the road that vehicle 220 is traveling on, it is desirable to detect lanes on that road in a consistent manner (by detecting either actual or artificially altered states). To this end, the developers of the present technology have developed solutions for selecting digital images to be used for retraining the ODNN so that it detects objects in use-phase digital images in a consistent manner.

[132] In general, processor 110 and/or server 235 are configured to select at least one of first digital image 300 and second digital image 500 for use in retraining the ODNN to perform more consistent object detection by comparing (a) the first data 490 usage stage detection associated with the first digital image 300, with (b) second usage stage detection data 690 associated with the second digital image 500.

[133] For example, processor 110 and/or server 235 may compare first portion 403 of first digital image 300 with second portion 603 of second digital image 500 to determine the degree of detection similarity for first object 400 (and modified first object 400'). The degree of similarity of detection indicates the degree of similarity of predictions when detecting the same real object (a specific lane of the road) performed by the ODNN network at the first and second moments of time.

[134] In some embodiments of the invention, the processor 110 and/or the server 235 in such a comparison may perform an IOU analysis on the first part 403 and the second part 603. In general, as a result of the IOU analysis, the IOU parameter is determined as the ratio (a) of the area the intersection of the first part 403 and the second part 603 (Fig. 6) to (b) the area of the union of the first part 403 (Fig. 4) and the second part 603.

[135] For example, the closer the IOU parameter is to "1", the more the localization of the first object 400 is similar to the localization of the modified first object 400'. Similarly, the closer the IOU parameter is to "0", the less the localization of the first object 400 is similar to the localization of the modified first object 400'. In some embodiments, the degree of similarity of detection determined by the processor 110 and/or the server 235 may be an IOU parameter for the first part 403 and the second part 603.

[136] In some embodiments of the present technology, it is contemplated that the IOU analysis can be performed on the projections of the first part 403 and the second part 603 onto a two-dimensional surface, instead of performing the IOU analysis directly on the first part 403 and the second part 603. This means that in In some embodiments of the present technology, processor 110 is capable of (a) projecting first part 403 onto a 2D surface, (b) projecting second part 603 onto a 2D surface, and (c) performing IOU analysis on the projections of first part 403 and second part 603 instead of doing IOU analysis directly for the first part 403 and the second part 603.

[137] The processor 110 and/or the server 235 may compare the degree of similarity of the detection with a predetermined threshold. The predetermined threshold may be determined by an operator of processor 110 and/or server 235. For example, an operator of processor 110 and/or server 235 may empirically determine the predetermined threshold. If the detection similarity value falls below a predetermined threshold, processor 110 may use at least one of first digital image 300 and second digital image 500 to retrain the ODNN.

[138] Assuming that the first digital image 300 is to be used, then the first digital image 300 may be provided to human evaluators tasked with identifying and locating objects of the first digital image 300. For example, they may be tasked with identifying a human-assessed location tag objects in the first digital image 300, as well as the corresponding classes of these objects (eg, generate a human-rated label for the first digital image 300). In some cases, they may be tasked with determining the location of objects in accordance with real road conditions. And in other cases, as explained above, they may be tasked with locating objects according to artificially altered road conditions.

[139] Processor 110 and/or server 235 may perform ODNN retraining based on at least one selected of first digital image 300 and second digital image 500 and/or corresponding human-rated label data. For example, a second stage of ODNN training may be performed by processor 110 and/or server 235 based on at least one of first digital image 300 and second digital image 500 and corresponding human-rated label data to provide conditions for object detection to the ODNN during the use phase. in a more consistent way.

[140] In some embodiments of the present technology, a method 700 for generating training data for retraining an ODNN is used, which can be performed by a computing device, such as a processor 110 and/or a server 235. The method 700 is described in more detail below.

Step 702: Entering the first digital image into the ODNN.

[141] Method 700 begins at step 702 by inputting a first digital image 300 into the ODNN by a computing device associated with the ODNN (eg, processor 110 and/or server 235). The first digital image 300 is the digital signal of the sensor 230 at the first time.

[142] The ODNN is capable of detecting a given object (eg, the first object 400 shown in FIG. 4) in the first digital image 300 by determining the first portion 403 of the first digital image 300 corresponding to that object.

[143] For example, prior to using the ODNN by the computing device in step 702, the ODNN may be trained in the training phase. As mentioned above, it is contemplated that in some embodiments of the present technology, an ODNN can be trained (up to step 702) to detect an object based on (a) a training digital image and (b) a human-rated label for objects in the training digital image, wherein the network ODNN predicts feature classes and feature locations on a training digital image.

Step 704: Entering a second digital image into the ODNN.

[144] Method 700 continues at step 704 with the computing device entering the second digital image 500 into the ODNN. The second digital image 500 is the digital signal of the sensor 230 at a second time point after the first time point.

[145] The ODNN is capable of detecting a given object (eg, a modified first object 400') in the second digital image 500 by determining the first portion 603 of the second digital image 500 corresponding to that object.

[146] It should be noted that both the modified first object 400' and the first object 400 detected by the ODNN correspond to the same real object (for example, a particular lane), however, the modified first object 400' and the first object 400 are localized by the ODNN network differently when they are discovered, i.e. ODNN defines different parts (eg, locations) of digital images corresponding to the same real object.

Step 706: comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity of object detection.

[147] Method 700 continues at step 706, where the computing device compares the first part 403 of the first digital image 300 with the second part 603 of the second digital image 500 to determine the degree of similarity of object detection.

[148] The degree of similarity of detection indicates the degree of similarity between the object detection predictions made by the ODNN at the first time and at the second time. You can also say that the degree of similarity of detection indicates how similar the predictions of the ODNN network are to each other regarding the location of this object in the corresponding digital images.

[149] In some embodiments of the present technology, comparing the first portion 403 of the first digital image 300 with the second portion 603 of the second digital image 500 to determine the degree of similarity of detection may include the computer device applying an IOU analysis. It is contemplated that applying the IOU analysis as above includes (a) determining the intersection of the first part 403 and the second part 603, and (b) determining the union of the first part 403 and the second part 603.

Step 708: using at least one of the first digital image and the second digital image to obtain a human-rated label while reducing the degree of similarity of detection below a predetermined threshold.

[150] The implementation of method 700 continues at step 708, where, as the degree of similarity of detection decreases below a predetermined threshold, the computing device uses at least one of the first digital image 300 and the second digital image 500 to obtain a human-assessed label indicating the real part at least one of the first digital image 300 and the second digital image 500 occupied by that object in the respective one of the first digital image 300 and the second digital image 500.

[151] In some embodiments, the actual portion occupied by an object can be identified by a human evaluator based on the actual state of that object, as described above. In other embodiments, the implementation of the real part of this object can be identified by a human appraiser based on an artificially changed state of this object, as described above.

STEP 710: retraining the ODNN based on at least one of the first digital image and the second digital image and/or a human-rated label.

[152] Method 700 continues at step 710, where the computing device retrains the ODNN based on at least one of the first digital image 300, the second digital image 500, and corresponding human-rated labels.

[153] For example, the first digital image 300 with a corresponding human-rated label may be used to retrain the ODNN. In another example, a second digital image 500 with a corresponding human-rated label may be used to retrain the ODNN. In yet another example, both a first digital image 300 with a corresponding human-rated label and a second digital image 500 with a corresponding human-rated label can be used to retrain the ODNN.

[154] It should be recalled that the real part for an object in a digital image, identified by a human evaluator, may depend on the task assigned to the human evaluator - to identify the actual state of this object or to identify an artificially altered state of this object. Thus, if a human evaluator is tasked with identifying a valid part for an object according to its actual state, the ODNN can be retrained to detect objects according to their actual state. If a human evaluator is tasked with identifying a real part of an object according to the artificially changed state, the ODNN can be retrained to detect objects according to the artificially changed state.

[155] In some embodiments of the present technology, the steps of method 700 described above may be repeated for a third digital image (not shown) from a digital sensor signal 230 and for a fourth digital image (not shown) from a digital sensor signal 230, similar to how a computer device performs the above steps for the first digital image 300 and the second digital image 500 within the spirit and scope of the present technology.

[156] Accordingly, it is contemplated that the computing device can generate a dataset for retraining the ODNN. For example, the retraining dataset may contain any combination of (a) a first digital image 300 with a corresponding human-rated label, (b) a second digital image 500 with a corresponding human-rated label, (c) a third digital image with a corresponding human-rated label, and ( d) a fourth digital image with a corresponding human-rated label.

[157] In some embodiments of the present technology, the computing device may determine that the area 250 around the vehicle 220 at the first time and/or at the second time is an inconsistent detection area. For example, vehicle 220 may include a GPS-type sensor that may provide location data indicative of the location of surrounding area 250 at a given point in time. As a result, if at least one of the first digital image 300 and the second digital image 500 is to be used to retrain the ODNN, the computing device may determine that the surrounding area 250 around the vehicle 220 at a given time when at least one from the first digital image 300 and the second digital image 500, is the area of inconsistent detection. Thus, in some embodiments, in addition to retraining the ODNN as described above, the computing device may flag and track areas of inconsistent detection for further analysis.

[158] It is contemplated that the retraining dataset generated for the ODNN may be used by a computing device (which may be, for example, the processor 110 of the electronic device 210 or the processor 110 of the server 235) to retrain the ODNN with digital image features, where object detection by the ODNN is inconsistent. As mentioned above, when the ODNN performs object detection based on two digital images, such as the first digital image 300 and the second digital image 500, the ODNN performs object detection based on similar inputs. However, as described above, despite the similarity of the input data (eg, data of the first digital image 300 and the second digital image 500), the output data of the ODNN may differ significantly. Thus, (a) generating a retraining dataset based on situations where similar ODNN inputs resulted in significantly different ODNN outputs, and (b) using this retraining dataset to retrain the ODNN can reduce the chance of inconsistent detection of objects by the ODNN network after the retraining stage. For example, the computing device can retrain the ODNN on the features of the first digital image 300 and/or the second digital image 500 (features obtained by the ODNN based on the corresponding image data), whereby after the retraining step, the probability of inconsistent object detection by the ODNN is reduced when receiving similar input data of the first digital image 300 and/or the second digital image 500.

[159] It should be noted that ODNN retraining by the computing device may be performed to reduce the likelihood of inconsistent detection of an object by the ODNN when vehicle 220 (or other vehicles implemented similarly to vehicle 220) is in an inconsistent detection region. When, after the ODNN retraining step, the digital image data captured in the inconsistent detection region is input to the ODNN, the ODNN can be less likely to repeat the inconsistent object detection. In other words, although before the retraining step, the ODNN may perform inconsistent object detection based on digital image data captured in the inconsistent detection region, after the retraining step, the ODNN will be less likely to perform inconsistent object detection based on digital image data. fixed in the area of inconsistent detection. Therefore, it can be said that the retraining of the ODNN as described above can be performed so that the ODNN is less likely to perform inconsistent object detection based on digital image features captured in the inconsistent detection region. Furthermore, it is contemplated that ODNN retraining can be performed to reduce the likelihood of inconsistent detection of objects by the ODNN based on features of digital images captured in regions like the inconsistent detection region. For example, digital images of areas like the area of mismatch may be associated with features similar to those of digital images of the area of mismatch.

[160] For a person skilled in the art, possible changes and improvements in the above-described embodiments of the present technology may be obvious. The foregoing description is for illustrative purposes only and is not intended to limit the scope of the invention. The scope of protection of this technology is determined solely by the scope of the appended claims.

Claims (58)

1. A method for generating training data for retraining an object detecting Neural Network (ODNN) trained to detect objects from a digital signal captured by a moving vehicle by determining a part of a digital image corresponding to these objects, performed by a computer device and including includes steps in which: - introducing into the ODNN network the first digital image, which is a digital signal at the first time point, wherein the ODNN is configured to detect an object from the first digital image by determining the first part of the first digital image corresponding to the object; - introducing a second digital image into the ODNN network, which is a digital signal at the second time moment after the first time moment, wherein the ODNN is configured to detect an object from the second digital image by determining the second part of the second digital image corresponding to the object; - comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity of object detection, indicating the degree of similarity of the predictions when detecting this object, performed by the ODNN network at the first time point and the second time point; - if the degree of similarity of detection is below a predetermined threshold value, - using at least one of the first digital image and/or the second digital image to obtain a human-rated mark indicating the actual part of at least one of the first digital image and/or the second digital image occupied by the object in the corresponding first digital image and/ or a second digital image; and re-training the ODNN based on at least one of the first digital image and the second digital image and/or the human-rated label. 2. The method of claim. 1, wherein comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity of the detection includes the use of IOU analysis (Intersection Over Union analysis). 3. The method of claim 2, wherein applying the IOU analysis includes determining the intersection of the first part and the second part, and determining the union of the first part and the second part. 4. The method of claim 1, further comprising receiving from the human appraiser an indication of the actual portion occupied by the object as determined by the human appraiser based on the actual state of the object. 5. The method of claim 1, further comprising receiving from the human appraiser an indication of the actual portion occupied by the object as determined by the human appraiser based on an artificially altered state of the object. 6. The method of claim 4, wherein the ODNN is retrained to detect objects according to the actual state of the objects. 7. The method of claim 5, wherein the ODNN is retrained to detect objects according to the artificially changed state of the objects. 8. The method of claim 1, wherein the ODNN is retrained based on the first digital image, the second digital image, and corresponding human-rated labels. 9. The method of claim 1, further comprising the steps of: - enter into the network ODNN the third digital image, which is a digital signal at the third point in time, while the network ODNN is able to detect an object in the third digital image by determining the third part of the third digital image corresponding to this object; - introducing into the ODNN network a fourth digital image, which is a digital signal at the fourth time point after the third time point, while the ODNN network is able to detect an object in the fourth digital image by determining the fourth part of the fourth digital image corresponding to this object; - comparing the third part of the third digital image with the fourth part of the fourth digital image to determine the degree of similarity of object detection, indicating the degree of similarity of the predictions when detecting this object, performed by the ODNN network at the third time point and the fourth time point; and - if the degree of similarity of detection is below a predetermined threshold value, - using at least one of the third digital image and the fourth digital image to obtain a human-rated mark indicating the actual part of at least one of the third digital image and the fourth digital image occupied by the object in the corresponding third digital image and/or fourth digital image . 10. The method of claim 9, wherein at least one of the first digital image and the second digital image with a corresponding human-rated label and at least one of the third digital image and the fourth digital image with a corresponding human-rated label form at least partially data set for retraining the ODNN network. 11. The method of claim 10, wherein the ODNN retraining dataset is used to retrain the ODNN on digital image features in which the ODNN object detection is inconsistent. 12. The method of claim 1, wherein the ODNN is trained to detect objects based on the training digital image and a human-rated label for objects in the training digital image, wherein the ODNN predicts object classes and object locations from the training digital image. 13. The method according to claim 9, wherein the area around the moving vehicle at at least one of: the first time point, the second time point, the third time point, the fourth time point is defined as the inconsistent detection area. 14. The method of claim 13, wherein ODNN retraining is performed to reduce the likelihood of inconsistent object detection by the ODNN when the vehicle is in an inconsistent detection region. 15. The method of claim 13, wherein ODNN retraining is performed to reduce the likelihood of inconsistent object detection by the ODNN based on digital image features captured in the inconsistent detection region. 16. The method of claim 13, wherein ODNN retraining is performed to reduce the likelihood of inconsistent detection of an object by the ODNN based on features of digital images captured in areas like the inconsistent detection area. 17. A computer device for generating training data for retraining an object detection neural network (ODNN) trained to detect objects from a digital signal captured by a moving vehicle by determining a part of the digital image corresponding to these objects, while the computer device is configured to: - input into the ODNN network of the first digital image, which is a digital signal at the first moment of time, wherein the ODNN is configured to detect an object in the first digital image by determining the first part of the first digital image corresponding to the object; - input into the ODNN network of the second digital image, which is a digital signal at the second time moment after the first time moment, wherein the ODNN is configured to detect an object in the second digital image by determining the second part of the second digital image corresponding to the object; - comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity of the detection of the object, indicating the degree of similarity of the predictions for the detection of this object, performed by the ODNN network at the first time and the second time; if the degree of similarity of detection is below a predetermined threshold value, - using at least one of the first digital image and the second digital image to obtain a human-rated mark indicating the real part of at least the world of one of the first digital image and the second digital image occupied by an object in the corresponding one of the first digital image and the second digital image ; and - retraining the ODNN based on at least one of the first digital image and the second digital image and the human-rated label. 18. The computer device of claim 17, further configured to apply IOU analysis by comparing the first part of the first digital image with the second part of the second digital image to determine the degree of similarity of the detection. 19. The computing device of claim 18, further configured to determine the intersection of the first part and the second part, and to determine the union of the first part and the second part when applying the IOU analysis. 20. The computer device of claim 17, further configured to receive from the human appraiser an indication of the actual portion occupied by the object as determined by the human appraiser based on the actual state of the object. 21. The computer device of claim 17, further configured to receive from the human appraiser an indication of the actual portion occupied by the object as determined by the human appraiser based on an artificially altered state of the object. 22. The computing device of claim 20, wherein ODNN retraining is performed to detect objects according to the actual state of the objects. 23. The computer device of claim 21, wherein retraining of the ODNN is performed to detect objects in accordance with the artificially changed state of the objects. 24. The computing device of claim 17, wherein the ODNN is retrained based on the first digital image, the second digital image, and corresponding human-rated labels. 25. The computer device according to claim 17, additionally configured to: - input to the ODNN network of the third digital image, which is a digital signal at the third point in time, while the ODNN network is able to detect an object in the third digital image by determining the third part of the third digital image corresponding to this object; - input to the ODNN network of the fourth digital image, which is a digital signal at the fourth time point after the third time point, while the ODNN network is able to detect an object in the fourth digital image by determining the fourth part of the fourth digital image corresponding to this object; - comparing the third part of the third digital image with the fourth part of the fourth digital image to determine the degree of similarity of the detection of the object, indicating the degree of similarity of the predictions when detecting this object, performed by the ODNN network at the third time point and the fourth time point; and if the degree of similarity of detection is below a predetermined threshold value, - using at least one of the third digital image and the fourth digital image to obtain a human-rated mark indicating the actual portion of at least one of the third digital image and the fourth digital image occupied by an object in the respective third digital image and/or fourth digital image . 26. The computing device of claim 25, wherein at least one of the first digital image and the second digital image with a corresponding human-rated label and at least one of the third digital image and the fourth digital image with a corresponding human-rated label form at least partially a dataset for retraining an ODNN network. 27. The computing device of claim 26, wherein the ODNN retraining dataset is used to retrain the ODNN on digital image features in which the ODNN object detection is inconsistent. 28. The computer device of claim 17, wherein the ODNN is trained to detect objects based on the training digital image and a human-rated label for objects in the training digital image, wherein the ODNN is capable of predicting object classes and object locations in the training digital image. 29. The computer device according to claim 25, wherein the area around the moving vehicle at at least one of: the first time point, the second time point, the third time point, the fourth time point is defined as the inconsistent detection area. 30. The computing device of claim 29, wherein ODNN retraining is performed to reduce the likelihood of inconsistent object detection by the ODNN when the vehicle is in an inconsistent detection region. 31. The computing device of claim 29, wherein ODNN retraining is performed to reduce the likelihood of inconsistent object detection by the ODNN based on digital image features captured in the inconsistent detection region. 32. The computer device of claim 29, wherein ODNN retraining is performed to reduce the likelihood of inconsistent detection of an object by the ODNN based on digital image features captured in areas like the inconsistent detection area.

Images