CN112990625A - Method and device for allocating annotation tasks and server - Google Patents
Method and device for allocating annotation tasks and server Download PDFInfo
- Publication number
- CN112990625A CN112990625A CN201911284516.2A CN201911284516A CN112990625A CN 112990625 A CN112990625 A CN 112990625A CN 201911284516 A CN201911284516 A CN 201911284516A CN 112990625 A CN112990625 A CN 112990625A
- Authority
- CN
- China
- Prior art keywords
- user
- labeling
- task
- category
- labeled
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06311—Scheduling, planning or task assignment for a person or group
Landscapes
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Operations Research (AREA)
- Game Theory and Decision Science (AREA)
- Development Economics (AREA)
- Marketing (AREA)
- Educational Administration (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a method and a device for allocating a labeling task and a server. The method comprises the following steps: acquiring the category interestingness of the labeling task of the user; respectively calculating the similarity between the category interestingness and a plurality of pictures to be labeled in a labeling database; and determining the picture to be annotated with the similarity meeting a preset threshold value as an annotation task of the user, and adding the annotation task into a task list of the user. According to the embodiment of the invention, the picture to be labeled with higher similarity to the category interestingness of the user can be taken as a labeling task to be distributed to the user, so that the accuracy and the efficiency of labeling are improved.
Description
Technical Field
The present invention relates to the field of image annotation technologies, and in particular, to an allocation method for an annotation task, an allocation apparatus for an annotation task, a server, and a computer-readable storage medium.
Background
In the computer vision recognition technology, a large amount of image data with labeled data is usually trained through a neural network algorithm to obtain a recognition model. Therefore, the accuracy of labeling the picture can affect the accuracy of identifying the model.
At present, most of the labeling tasks of the pictures are randomly distributed to labeling personnel, and the labeling personnel label the pictures in a manual labeling mode. However, due to subjective differences such as cognitive abilities and interest preferences of annotating personnel, accuracy and efficiency of annotating pictures can be affected to a certain extent.
Therefore, there is a need to provide a new method for assigning annotation tasks to improve the accuracy and efficiency of annotating pictures.
Disclosure of Invention
It is an object of embodiments of the present invention to provide a new solution for allocation of annotation tasks.
According to a first aspect of the present invention, there is provided an annotation task allocation method, the method comprising:
acquiring the category interestingness of the labeling task of the user;
respectively calculating the similarity between the category interestingness and a plurality of pictures to be labeled in a labeling database;
and determining the picture to be marked with the similarity meeting a preset threshold value as a marking task of the user, and adding the marking task into a task list of the user.
Optionally, the obtaining of the category interestingness of the annotation task of the user includes:
calculating the marking accuracy of the user to the picture to be marked of each category;
obtaining the interest score of the labeling result of each user according to the labeling accuracy of each category and the number of the pictures to be labeled of the corresponding category;
and determining the category interestingness of the labeling task of the user according to the interest score.
Optionally, the determining the category interestingness of the annotation task of the user according to the interest score includes:
determining the category with the highest interest score as the labeling interest point of the user;
randomly selecting a preset number of pictures to be labeled from the categories corresponding to the labeling interest points, calculating the median of the feature vectors of the selected pictures to be labeled, and determining the median of the feature vectors as the category interestingness of the labeling task of the user.
Optionally, the respectively calculating the similarity between the category interestingness and the plurality of pictures to be labeled in the labeling database includes:
and respectively calculating the cosine distance between the category interestingness and the characteristic value of each picture to be labeled.
Optionally, before determining the to-be-annotated picture whose similarity meets a preset threshold as an annotation task of the user, the method further includes:
determining the task quantity of the labeling task of each user according to the total quantity of the pictures to be labeled and the number of the users;
dividing the task quantity of the labeling tasks into a first quantity and a second quantity according to a preset proportion;
correspondingly, the determining the to-be-labeled picture with the similarity meeting the preset threshold as the labeling task of the user includes:
and determining the first number of pictures to be labeled, the similarity of which meets the preset threshold value, and the randomly selected second number of pictures to be labeled as the labeling tasks of the user.
Optionally, before the obtaining of the category interestingness of the annotation task of the user, the method further includes:
judging whether the category interestingness of the user exists or not;
if yes, executing the operation of acquiring the category interestingness of the labeling task of the user;
and if not, randomly selecting the picture to be labeled as the labeling task of the user and adding the selected picture to the task list of the user.
Optionally, the method further comprises:
and when the pictures to be labeled are imported into the labeling database, extracting the characteristic vector of each picture to be labeled through a neural network algorithm.
According to a second aspect of the present invention, there is provided an annotation task allocation apparatus, comprising:
the acquisition module is used for acquiring the category interestingness of the labeling task of the user;
the calculation module is used for respectively calculating the similarity between the category interestingness and a plurality of pictures to be labeled in the labeling database;
and the distribution module is used for determining the pictures to be labeled, the similarity of which meets a preset threshold value, as the labeling tasks of the user and adding the labeling tasks into a task list of the user.
Optionally, the obtaining module is specifically configured to:
calculating the marking accuracy of the user to the picture to be marked of each category;
obtaining the interest score of the labeling result of each user according to the labeling accuracy of each category and the number of the pictures to be labeled of the corresponding category;
and determining the category interestingness of the labeling task of the user according to the interest score.
Optionally, the obtaining module is specifically configured to:
determining the category with the highest interest score as the labeling interest point of the user;
randomly selecting a preset number of pictures to be labeled from the categories corresponding to the labeling interest points, calculating the median of the feature vectors of the selected pictures to be labeled, and determining the median of the feature vectors as the category interestingness of the labeling task of the user.
Optionally, the calculation module is specifically configured to:
and respectively calculating the cosine distance between the category interestingness and the characteristic value of each picture to be labeled.
Optionally, the allocation module is further configured to determine a task amount of the labeling task of each user according to the total amount of the pictures to be labeled and the number of the users; dividing the task quantity of the labeling tasks into a first quantity and a second quantity according to a preset proportion; and determining the first number of pictures to be labeled, the similarity of which meets the preset threshold value, and the randomly selected second number of pictures to be labeled as the labeling tasks of the user.
Optionally, the device for allocating an annotation task further includes a determining module, configured to determine whether a category interest degree of the user exists;
if the user's annotation task exists, triggering the acquisition module to execute the operation of acquiring the category interestingness of the user's annotation task;
if the image to be annotated does not exist, the distribution module is triggered to randomly select the image to be annotated as the annotation task of the user and add the image to the task list of the user.
Optionally, the obtaining module is further configured to extract a feature vector of each picture to be labeled through a neural network algorithm when the picture to be labeled is imported into the labeling database.
According to a third aspect of the present invention, there is provided a server comprising the annotating task assigning device according to the second aspect of the present invention, or the server comprises:
a memory for storing executable commands;
a processor for executing the method of assigning annotation tasks according to any one of the first aspect of the invention under the control of the executable command.
According to a fourth aspect of the present invention, there is provided a computer readable storage medium storing executable instructions which, when executed by a processor, perform the method of assigning annotation tasks according to any one of the first aspect of the present invention.
According to one embodiment of the invention, the picture to be labeled with higher similarity to the user category interestingness can be distributed to the user as a labeling task, so that the accuracy and efficiency of labeling are improved.
Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a schematic structural diagram of a server to which the method for allocating annotation tasks according to the embodiment of the present invention can be applied;
FIG. 2 is a flow chart of a method of assignment of annotation tasks in accordance with an embodiment of the present invention;
FIG. 3 shows a schematic flow diagram of an example according to an embodiment of the invention;
FIG. 4 is a schematic structural diagram of a task labeling distribution device according to an embodiment of the present invention;
FIG. 5 is a functional block diagram of a server according to an embodiment of the present invention.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
< hardware configuration >
Fig. 1 is a block diagram showing a hardware configuration of a server 1000 that can implement an embodiment of the present invention.
Server 1000 may be, for example, a blade server or the like.
In one example, server 1000 may be a computer.
In another example, the server 1000 may be as shown in fig. 1, including a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600. Although the server may also include speakers, microphones, etc., these components are not relevant to the present invention and are omitted here.
The processor 1100 may be, for example, a central processing unit CPU, a microprocessor MCU, or the like. The memory 1200 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a serial interface, and the like. Communication device 1400 is capable of wired or wireless communication, for example. The display device 1500 is, for example, a liquid crystal display panel. The input device 1600 may include, for example, a touch screen, a keyboard, and the like.
The servers shown in fig. 1 are merely illustrative and are in no way meant to limit the invention, its application, or uses. In an embodiment of the present invention, the memory 1200 of the server 1000 is used for storing instructions for controlling the processor 1100 to operate so as to execute any one of the annotated tasks allocation methods provided by the embodiment of the present invention.
It should be understood by those skilled in the art that although a plurality of devices are shown for the server 1000 in fig. 1, the present invention may only relate to some of the devices, for example, only the processor 1100 and the storage device 1200 of the server 1000.
The skilled person can design the instructions according to the disclosed solution. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.
< method examples >
The embodiment provides an allocation method of annotation tasks, which may be implemented by a server, for example, the server may be the server 1000 shown in fig. 1.
As shown in FIG. 2, the method comprises the following steps 2100 to 2300:
in step 2100, the category interestingness of the annotation task of the user is obtained.
The user refers to a label person, and the category interestingness refers to the cognitive accuracy of the user on a certain category of pictures. For example, a user likes a car at ordinary times, and knows the model, style, and the like of each brand of car, so that the recognition accuracy of the picture of the car class by the user is correspondingly higher.
It should be noted that, if the user has not accepted the annotation task in the annotation system, the annotation system does not have the category interest level of the user. That is, the category interestingness is calculated based on the accuracy of the annotation task that the user has historically performed.
Specifically, when calculating the category interestingness of the user, the server 1000 may first obtain the labeling result of the labeling task of the user; and then, carrying out manual quality inspection on the labeling result to obtain a quality inspection result.
Further, when calculating the category interestingness of the user based on the quality inspection result, the server 1000 determines the labeled interest point of the user, and then determines the category interestingness of the user based on the labeled interest point.
Specifically, when determining the labeling interest point of the user, the server 1000 calculates the labeling accuracy of the user for each category of the to-be-labeled picture according to the quality inspection result; obtaining the interest score of the labeling result of each user according to the labeling accuracy of each category and the number of the pictures to be labeled of the corresponding category; and determining the category interestingness of the labeling task of the user according to the interest score.
When determining the category interestingness of the annotation task of the user according to the interest score, the server 1000 may specifically determine the category with the highest interest score as the annotation interest point of the user; randomly selecting a preset number of pictures to be labeled from the categories corresponding to the labeling interest points, calculating the median of the feature vectors of the selected pictures to be labeled, and determining the median of the feature vectors as the category interestingness of the labeling task of the user.
It can be understood that if the annotation interest points of different users conflict, the annotation interest points of the users with lower scores move forward to the next category.
In practical application, each time a user completes a labeling task, a corresponding quality inspection result is obtained, and the server 1000 repeats the operation of calculating the category interest degree of the user, so that the calculated category interest degree of the user is more and more accurate, and thus, the labeling task allocated to the user is more suitable for the user, and further, the accuracy rate and the labeling speed of the user for labeling pictures are higher and higher.
Based on this, in practical application, after the user logs in the annotation system, the server 1000 may first determine whether the annotation system has the category interestingness of the user before acquiring the category interestingness of the annotation task of the user; if yes, executing the operation of acquiring the category interestingness of the labeling task of the user in the step; if the image to be annotated does not exist, the server 1000 randomly selects the image to be annotated as the annotation task of the user, and adds the selected image to the task list of the user.
After the category interestingness of the annotation task of the user is obtained, step 2200 is entered:
in step 2200, the similarity between the category interestingness and the plurality of pictures to be labeled in the labeling database is calculated respectively.
In practical applications, when the server 1000 imports the to-be-labeled picture into the labeling database, the feature vector of each to-be-labeled picture is extracted through a neural network algorithm, that is, each to-be-labeled picture is represented by a high-dimensional feature vector.
In this step, after obtaining the category interestingness of the user, the server 1000 performs retrieval calculation in the annotation database, and respectively calculates the cosine distance between the category interestingness and the feature value of each to-be-annotated picture. In some other possible implementation manners, when the server 1000 calculates the similarity between the category interestingness and the feature value of each to-be-labeled picture, other metric values used for representing the similarity may also be used, which is not specifically limited in this embodiment.
After calculating the similarity, step 2300 is entered:
in step 2300, the to-be-annotated picture whose similarity meets the preset threshold is determined as the annotation task of the user, and added to the task list of the user.
When allocating the annotation task, the server 1000 first needs to determine the task amount of the annotation task for each user according to the total amount of the pictures to be annotated and the number of the users. For example, the total number of pictures to be annotated is T, and the number of users is P, the task amount of the annotation task for each user should be N ═ T/P.
In an example, the server 1000 may determine, as the annotation task allocated to the user, N to-be-annotated pictures whose cosine distances meet a preset threshold.
In another example, in order to balance the category interestingness of each user, each time the server 1000 assigns an annotation task, a part of the pictures to be annotated of other categories may be randomly assigned to the user for annotation. That is, the server 1000 may divide the task amount of the annotation task into a first amount and a second amount according to a preset proportion; and determining the first number of the pictures to be labeled, the similarity of which meets the preset threshold value, and the randomly selected second number of the pictures to be labeled as the labeling tasks of the user.
For example, assuming that the preset proportion is 50%, the server 1000 may divide the task amount of the annotation task into a first number K of N/2 and a second number M of N/2, that is, select K to-be-annotated pictures meeting the preset threshold and M to-be-annotated pictures selected randomly as the annotation tasks allocated to the user for annotation.
It is understood that, after the to-be-annotated pictures with similarity meeting the preset threshold value in the annotation database are allocated to the user, the server 1000 marks the to-be-annotated pictures as allocated in the annotation database, so as to prevent the to-be-annotated pictures from being repeatedly allocated to other users.
< example >
Fig. 3 shows a schematic flow diagram of an example according to an embodiment of the invention.
As shown in fig. 3, the method for allocating annotation tasks in this embodiment may include the following steps:
and 3100, importing the picture to be annotated into an annotation database.
When the pictures to be labeled are imported, extracting the characteristic vector of each picture to be labeled through a neural network algorithm.
And 3300, obtaining a category interestingness of the labeling task of the user.
And 3400, randomly selecting the picture to be annotated as the annotation task of the user. Step 3800 is then performed.
And 3600, determining the task quantity of the labeling task of each user according to the total quantity of the pictures to be labeled and the number of the users.
3700, dividing the task quantity of the labeling task into a first quantity and a second quantity according to a preset proportion, determining the first quantity of the to-be-labeled pictures with the similarity meeting the preset threshold and the second quantity of the to-be-labeled pictures randomly selected as the labeling task of the user, and waiting for the user to label.
And 3900, calculating the marking accuracy of the user for the to-be-marked picture of each category according to the quality inspection result.
3910, obtaining the interest score of the labeling result of each user according to the labeling accuracy of each category and the number of the pictures to be labeled of the corresponding category; and determining the category with the highest interest score as the labeled interest point of the user.
After the category interestingness of the annotation task of the user is obtained, the category interestingness of the user is updated to the annotation system, and when a new annotation task is allocated, the similarity between the picture to be annotated and the category interestingness of the user can be comprehensively considered, and then the annotation task is allocated to the user, that is, the operation in the step 3200 is executed. Therefore, the accuracy and the marking speed of the user marking are improved.
The assignment method of the labeling tasks in the present embodiment has been described above with reference to the drawings and examples. The method of the embodiment obtains the category interestingness of the labeling task of the user; respectively calculating the similarity between the category interestingness and a plurality of pictures to be labeled in a labeling database; and determining the picture to be annotated with the similarity meeting a preset threshold value as an annotation task of the user, and adding the annotation task into a task list of the user. Because the labeling task allocated to the user is closely related to the category interestingness of the user, the accuracy and the labeling speed of the user for labeling the pictures to be labeled of the category are higher, and the accuracy and the efficiency of labeling can be improved.
< apparatus embodiment >
The present embodiment provides an annotated task allocation apparatus, which is, for example, the annotated task allocation apparatus 4000 shown in fig. 4.
As shown in FIG. 4, the assignment device 4000 for labeling tasks may include an acquisition module 4100, a calculation module 4200 and an assignment module 4300.
The obtaining module 4100 is configured to obtain a category interestingness of an annotation task of a user;
the calculation module 4200 is configured to calculate similarities between the category interestingness and a plurality of to-be-labeled pictures in the labeling database, respectively;
the distribution module 4300 is configured to determine the to-be-annotated picture whose similarity meets a preset threshold as an annotation task of the user, and add the annotation task to a task list of the user.
In an example, the obtaining module 4100 is specifically configured to calculate the labeling accuracy of the user for each category of pictures to be labeled; obtaining the interest score of the labeling result of each user according to the labeling accuracy of each category and the number of the pictures to be labeled of the corresponding category; and determining the category interestingness of the labeling task of the user according to the interest score.
Specifically, the obtaining module 4100 may determine the category with the highest interest score as the labeled interest point of the user; randomly selecting a preset number of pictures to be labeled from the categories corresponding to the labeling interest points, calculating the median of the feature vectors of the selected pictures to be labeled, and determining the median of the feature vectors as the category interestingness of the labeling task of the user.
Specifically, the calculating module 4200 can be configured to calculate cosine distances between the category interestingness and the feature values of the to-be-labeled pictures respectively.
In an example, the allocation module 4300 may be further configured to determine a task amount of a labeling task of each user according to the total amount of the pictures to be labeled and the number of the users; dividing the task quantity of the labeling task into a first quantity and a second quantity according to a preset proportion; and determining the first number of the pictures to be labeled, the similarity of which meets the preset threshold value, and the randomly selected second number of the pictures to be labeled as the labeling tasks of the user.
In another example, the assigning apparatus 4000 for labeling task may further include a determining module, configured to determine whether there is a category interest of the user; if yes, triggering the obtaining module 4100 to execute the operation of obtaining the category interest degree of the labeling task of the user; if not, triggering the allocation module 4300 to randomly select the to-be-annotated picture as the annotation task of the user, and adding the to-be-annotated picture to the task list of the user.
In another example, the obtaining module 4100 can be further configured to extract a feature vector of each picture to be labeled through a neural network algorithm when the picture to be labeled is imported into the labeling database.
The assignment device of the labeling task of this embodiment may be configured to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects thereof are similar, and are not described herein again.
< apparatus embodiment >
In this embodiment, an electronic device is further provided, where the electronic device includes an allocation apparatus 4000 for labeling tasks described in the embodiment of the apparatus of the present invention; alternatively, the electronic device is the server 5000 shown in fig. 5, and includes:
a memory 5100 for storing executable commands.
The processor 5200 is configured to execute the methods described in any of the method embodiments of the present invention under the control of executable commands stored in the memory 5100.
< computer-readable storage Medium embodiment >
The present embodiments provide a computer-readable storage medium having stored therein executable instructions that, when executed by a processor, perform the method described in any of the method embodiments of the present invention.
The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.
Claims (10)
1. A method for assigning annotation tasks, the method comprising:
acquiring the category interestingness of the labeling task of the user;
respectively calculating the similarity between the category interestingness and a plurality of pictures to be labeled in a labeling database;
and determining the picture to be marked with the similarity meeting a preset threshold value as a marking task of the user, and adding the marking task into a task list of the user.
2. The method of claim 1, wherein the obtaining of the category interestingness of the annotation task of the user comprises:
calculating the marking accuracy of the user to the picture to be marked of each category;
obtaining the interest score of the labeling result of each user according to the labeling accuracy of each category and the number of the pictures to be labeled of the corresponding category;
and determining the category interestingness of the labeling task of the user according to the interest score.
3. The method of claim 2, wherein determining the category interestingness of the annotation task of the user based on the interest score comprises:
determining the category with the highest interest score as the labeling interest point of the user;
randomly selecting a preset number of pictures to be labeled from the categories corresponding to the labeling interest points, calculating the median of the feature vectors of the selected pictures to be labeled, and determining the median of the feature vectors as the category interestingness of the labeling task of the user.
4. The method according to claim 1, wherein the calculating the similarity between the category interestingness and the plurality of pictures to be labeled in the labeling database comprises:
and respectively calculating the cosine distance between the category interestingness and the characteristic value of each picture to be labeled.
5. The method according to claim 1, wherein before the determining that the picture to be annotated whose similarity meets the preset threshold is the annotation task of the user, the method further comprises:
determining the task quantity of the labeling task of each user according to the total quantity of the pictures to be labeled and the number of the users;
dividing the task quantity of the labeling tasks into a first quantity and a second quantity according to a preset proportion;
correspondingly, the determining the to-be-labeled picture with the similarity meeting the preset threshold as the labeling task of the user includes:
and determining the first number of pictures to be labeled, the similarity of which meets the preset threshold value, and the randomly selected second number of pictures to be labeled as the labeling tasks of the user.
6. The method of claim 1, wherein prior to obtaining the category interestingness of the user's annotation task, the method further comprises:
judging whether the category interestingness of the user exists or not;
if yes, executing the operation of acquiring the category interestingness of the labeling task of the user;
and if not, randomly selecting the picture to be labeled as the labeling task of the user and adding the selected picture to the task list of the user.
7. The method of claim 1, further comprising:
and when the pictures to be labeled are imported into the labeling database, extracting the characteristic vector of each picture to be labeled through a neural network algorithm.
8. An apparatus for assigning annotation tasks, the apparatus comprising:
the acquisition module is used for acquiring the category interestingness of the labeling task of the user;
the calculation module is used for respectively calculating the similarity between the category interestingness and a plurality of pictures to be labeled in the labeling database;
and the distribution module is used for determining the pictures to be labeled, the similarity of which meets a preset threshold value, as the labeling tasks of the user and adding the labeling tasks into a task list of the user.
9. A server comprising the annotating task assigning device according to claim 8, or comprising:
a memory for storing executable commands;
a processor for performing the method of assigning annotation tasks according to any of claims 1 to 7 under the control of the executable command.
10. A computer-readable storage medium, having stored thereon executable instructions, which when executed by a processor, perform the method of assigning annotation tasks according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911284516.2A CN112990625A (en) | 2019-12-13 | 2019-12-13 | Method and device for allocating annotation tasks and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911284516.2A CN112990625A (en) | 2019-12-13 | 2019-12-13 | Method and device for allocating annotation tasks and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112990625A true CN112990625A (en) | 2021-06-18 |
Family
ID=76342115
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911284516.2A Pending CN112990625A (en) | 2019-12-13 | 2019-12-13 | Method and device for allocating annotation tasks and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112990625A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113449142A (en) * | 2021-06-30 | 2021-09-28 | 北京百度网讯科技有限公司 | Information processing method and device, electronic equipment, storage medium and product |
| CN113592270A (en) * | 2021-07-22 | 2021-11-02 | 上海淇玥信息技术有限公司 | Service voice annotation processing method and device and electronic equipment |
| CN113641838A (en) * | 2021-08-11 | 2021-11-12 | 上海明略人工智能(集团)有限公司 | Device and method for data annotation, electronic equipment and readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108829652A (en) * | 2018-04-28 | 2018-11-16 | 河海大学 | A kind of picture labeling system based on crowdsourcing |
| CN109829397A (en) * | 2019-01-16 | 2019-05-31 | 创新奇智(北京)科技有限公司 | A kind of video labeling method based on image clustering, system and electronic equipment |
| CN109919252A (en) * | 2019-03-26 | 2019-06-21 | 中国科学技术大学 | The method for generating classifier using a small number of mark images |
| CN109978356A (en) * | 2019-03-15 | 2019-07-05 | 平安普惠企业管理有限公司 | Mark method for allocating tasks, device, medium and computer equipment |
| CN110147824A (en) * | 2019-04-18 | 2019-08-20 | 微梦创科网络科技（中国）有限公司 | A kind of automatic classification method and device of image |
| CN110490444A (en) * | 2019-08-13 | 2019-11-22 | 新华智云科技有限公司 | Mark method for allocating tasks, device, system and storage medium |
-
2019
- 2019-12-13 CN CN201911284516.2A patent/CN112990625A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108829652A (en) * | 2018-04-28 | 2018-11-16 | 河海大学 | A kind of picture labeling system based on crowdsourcing |
| CN109829397A (en) * | 2019-01-16 | 2019-05-31 | 创新奇智(北京)科技有限公司 | A kind of video labeling method based on image clustering, system and electronic equipment |
| CN109978356A (en) * | 2019-03-15 | 2019-07-05 | 平安普惠企业管理有限公司 | Mark method for allocating tasks, device, medium and computer equipment |
| CN109919252A (en) * | 2019-03-26 | 2019-06-21 | 中国科学技术大学 | The method for generating classifier using a small number of mark images |
| CN110147824A (en) * | 2019-04-18 | 2019-08-20 | 微梦创科网络科技（中国）有限公司 | A kind of automatic classification method and device of image |
| CN110490444A (en) * | 2019-08-13 | 2019-11-22 | 新华智云科技有限公司 | Mark method for allocating tasks, device, system and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113449142A (en) * | 2021-06-30 | 2021-09-28 | 北京百度网讯科技有限公司 | Information processing method and device, electronic equipment, storage medium and product |
| CN113592270A (en) * | 2021-07-22 | 2021-11-02 | 上海淇玥信息技术有限公司 | Service voice annotation processing method and device and electronic equipment |
| CN113641838A (en) * | 2021-08-11 | 2021-11-12 | 上海明略人工智能(集团)有限公司 | Device and method for data annotation, electronic equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109034069B (en) | Method and apparatus for generating information | |
| CN109614238B (en) | Target object identification method, device and system and readable storage medium | |
| WO2019021088A1 (en) | Navigating video scenes using cognitive insights | |
| CN112990625A (en) | Method and device for allocating annotation tasks and server | |
| CN112749344A (en) | Information recommendation method and device, electronic equipment, storage medium and program product | |
| CN111523021B (en) | Information processing system and execution method thereof | |
| US9881000B1 (en) | Avoiding sentiment model overfitting in a machine language model | |
| CN110674620A (en) | Target file generation method, device, medium and electronic equipment | |
| US20170185964A1 (en) | Methods and systems for matching candidates and job positions bi-directionally using cognitive computing | |
| CN112559800A (en) | Method, apparatus, electronic device, medium, and product for processing video | |
| CN111582477A (en) | Training method and device of neural network model | |
| CN109034199B (en) | Data processing method and device, storage medium and electronic equipment | |
| US11423219B2 (en) | Generation and population of new application document utilizing historical application documents | |
| CN109684566B (en) | Label engine implementation method and device, computer equipment and storage medium | |
| CN113361240B (en) | Method, apparatus, device and readable storage medium for generating target article | |
| CN114418398A (en) | Scene task development method, device, equipment and storage medium | |
| CN114360027A (en) | Training method and device for feature extraction network and electronic equipment | |
| CN114330249A (en) | Information editing method, device, equipment and storage medium | |
| CN109034085B (en) | Method and apparatus for generating information | |
| CN113343700B (en) | Data processing method, device, equipment and storage medium | |
| US20220164534A1 (en) | Techniques for selecting content to include in user communications | |
| CN110580648A (en) | financial information processing method and device based on artificial intelligence | |
| CN114880498A (en) | Event information display method and device, equipment and medium | |
| CN109857838B (en) | Method and apparatus for generating information | |
| CN113807436A (en) | User mining method and device, computer equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |