US20090203413A1

US20090203413A1 - System and method for conducting competitions

Info

Publication number: US20090203413A1
Application number: US12/030,684
Authority: US
Inventors: Anthony Jefts; Gregory Eldridge; Javier Fernandez-Ivern; Michael A. Fogleman; John M. Hughes; David Messinger; George Tsipolitis
Original assignee: Individual
Current assignee: Topcoder LLC
Priority date: 2008-02-13
Filing date: 2008-02-13
Publication date: 2009-08-13

Abstract

In general, in one aspect, a method for developing an asset by competition includes posting a list of competitions, each competition for the development of an asset, each element of the list including a reference to one of a number of workspaces, the one of the workspaces allocated to a respective asset. The method includes providing a draft competition specification in one of the workspaces. The method includes facilitating feedback from potential competitors on the draft competition specification in the workspace for a period of time prior to the competition. The method includes finalizing the draft competition specification based on the feedback from potential competitors, and holding a competition for the development of the asset.

Description

TECHNICAL FIELD

This invention relates to computer-based methods and systems for facilitating the development of an asset.

BACKGROUND INFORMATION

Competitions have been held for the development of an asset, such as essays, music, computer software, and so on.
In some competitions, a larger work is divided up into a number of smaller pieces, and competitions held for the development of those pieces. For example, a software program may be divided up into components, and competitions held for the development of each of the components. In some cases, development of a component may be divided into design tasks and coding tasks, and a contest held for the design of the component, and another contest held for the coding of the component. As another example, a competition may be held for the development of a graphic design, such as a logo.
In such competitions, specifications typically are not made available until the competition begins. In such cases, specifications are developed without the benefit of input from potential competitors. Sometimes substantial interaction with the competitors during the competition is required to clarify parts of the specification that are not clear. Correction to specifications may be necessary in order to fix problems identified by the competitors, potentially requiring increases to the competition time and reducing the likelihood of competition success.

SUMMARY OF THE INVENTION

Various asset development competitions have been used to develop assets, such as content, computer software, graphics, user interfaces, and so forth. Such competitions have been held, for example, by TOPCODER, INC. of Glastonbury, Conn., and may be seen at http://www.topcoder.com.
As described in co-pending U.S. patent application Ser. No. 11/035,783, entitled SYSTEMS AND METHODS FOR SOFTWARE DEVELOPMENT by John M. Hughes, filed Jan. 14, 2005, incorporated herein by reference, a software application may be developed by contest, where the task of developing the software application is divided up into discrete tasks, and competitions are held for the performance of each of the discrete tasks. For example, a software program may be divided up into components, and competitions held for the development of each of the components. Development of a component may be divided into design tasks and coding tasks, and a contest held for the design of the component, and another contest held for the coding of the component.
As described in co-pending U.S. patent application Ser. No. 11/655,768, entitled SYSTEM AND METHOD FOR DESIGN DEVELOPMENT by John M. Hughes, filed Jan. 19, 2007, incorporated herein by reference, design contests may be held, for example, for graphics design and user interface development as well as other types of designs and work product. Submissions in such contests may be evaluated for technical merit (i.e., meeting the described requirements) and/or based on customer affinity and/or appeal to a designated group of individuals.
As described in co-pending U.S. patent application Ser. No. 11/755,909, entitled PROJECT MANAGEMENT SYSTEM, by Messinger et al., filed May 31, 2007, incorporated herein by reference, a series of contests may be managed through use of project management system. Portions of a contest, such as a review effort, may begin when submissions are received, deadlines are met, and so on. Such contest systems can provide contest-related information to contest administrators as well as potential competitors.
In various embodiments, techniques may be used to inform potential contestants in advance about upcoming competitions. At the same time, specifications may be made available to potential contestants, for review and comment. Making the specifications available in advance enables potential competitors to make better decisions about whether to participate in a competition, and also allows them to comment on or clarify the specification for the competition in advance of the competition.
For example, in some embodiments, a list of competitions is provided. Some of the competitions in the list may be upcoming. In addition, some of the competitions in the list may be active competitions and/or completed competitions. The list may be sorted and/or filtered by date, type of competition and/or other information. The list may be communicated using any suitable information communication technique, for example by one or more of a web page, email, RSS feed, text message, and so on. Each competition is for the development of one or more assets. Each competition listing may include a reference to one of a number of workspaces, typically a workspace allocated to the respective asset. For example, the workspace may be or may include one or more of a shared document, a wiki page, a web page, a blog page, a forum, a bulletin board, and so on. The workspace includes a place for providing a draft competition specification.
The competition specification may include a technical description of the asset to be developed. The competition specification may include the rules and/or parameters of the competition (e.g., dates, prizes, and so on). The competition specification may include other information, such as other documents and tools that may be used, reference information, submission details, competition logistics, and so on.
Feedback may be received from potential competitors on the draft competition specification in the workspace. Typically, feedback is received for a period of time prior to the competition. The feedback may include (among other things) any one or more of comments on the specification, text changes to the specification, graphical or design changes, other modifications to the specification, additions to the specification, and so on. The draft competition specification (including without limitation information about competition logistics) may be finalized based on the feedback, and a competition for the development of the asset held using the finalized specification as the competition specification.
The asset that is to be developed by competition may be any type of asset. Just as some non-limiting examples, the asset may be or may include software component design, graphic design, and mechanical part design, software component, product design, product specification or plans, creative work, artistic work, video, audio, multimedia presentation, music, etc. The list of competitions may include, for example, a competition name, a planned competition start date, other pertinent meta data describing the competition such as an overview, technologies involved, timelines, and so on and/or a reference to the workspace. The reference to the workspace may include a URL.
In general, in another aspect, a system for developing an asset by competition comprises a competition list including a list of competitions, each competition for the development of an asset, each element of the list including a reference to one of a number of workspaces, the one of the workspaces allocated to a respective asset. The system includes a workspace including a draft competition specification and a facility for receiving feedback from potential competitors on the draft competition specification in the workspace for a period of time prior to the competition. The system includes a competition system for holding a competition for the development of the asset using a specification that is based on the draft competition specification taking into account the feedback from potential competitors.
The systems described can be implemented as software running on computers, mobile devices and telephones, televisions, home entertainment centers, gaming consoles, and other devices, for example, as described in the above-referenced co-pending patent applications.
Other aspects and advantages of the invention will become apparent from the following drawings, detailed description, and claims, all of which illustrate the principles of the invention, by way of example only.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.

FIG. 1 is a flowchart illustrating by example a method for developing work product according to an embodiment of the invention.

FIG. 2 is a flow chart illustrating by example a method for determining an estimated likelihood of success according to an embodiment of the invention.

FIG. 3 is a block diagram illustrating by example an embodiment of a development contest.

FIG. 4 is a block diagram illustrating by example an embodiment of a contest-based software application development process.

FIG. 5 is a demonstrative screen display in an exemplary embodiment of the invention.

FIG. 6 is a demonstrative screen display in an exemplary embodiment of the invention.

DETAILED DESCRIPTION

Referring to FIG. 1, in some embodiments, a method 100 for developing an asset by competition may be applicable to any type of asset. Just as examples, not intended to be limiting, in various embodiments, the asset developed by competition may be or may include text, content, graphics, design, development, data, architecture, and/or specifications, for software, hardware, audio, video, music, machines, equipment, devices, and/or media. The competition may be any suitable competition. Just as examples, not intended to be limiting, the competition may be an on-line competition, in-person competition, live competition, and/or timed competition. In some embodiments, the competition is an on-line competition for the development of a portion of a computer software application, such as a design for computer code, computer code based on a design, or a user interface design.
Typically, a competition involves two or more participants competing to develop an asset. A specification describes requirements for an asset and may also include competition rules and logistics. Standards may be specified for the asset and may be specific to the type of asset. Standards may be described for example, in a checklist or scorecard, and may be general or specific as appropriate. In some embodiments, a review of an asset is conducted using the specification and the standards in order to identify a competition winner, and the review criteria include the relevant standards.
In various embodiments, techniques may be used to enable communication between potential contestants, contest administrators, and contest sponsors in advance of upcoming asset development competitions. In some cases, such techniques allow contestants to be informed about the logistics of upcoming competitions, so as to allow them to plan their schedules to allow participation. Such techniques allow contestants to provide feedback about the logistics, for example, to point out potential conflicts with other competitions or events, or to otherwise give an early indication that the planned logistics may negatively or positively affect the success of the competition.
In various embodiments, specifications may be made available to potential contestants in advance of the competitions for review and comment. Making the specifications available in advance enables potential competitors to make better decisions about whether to participate in a competition, and also allows them to comment on or clarify the specification for the competition in advance of the competition. The feedback received may be used to modify the specification or otherwise clarify the specification, to make the competition complete more smoothly. The feedback received also may be used to determine interest in the competition, or otherwise determine the success of the competition.
For example, in some embodiments, a list of competitions is provided 101. At least some of the competitions are upcoming and have not yet started. In some embodiments, some of the competitions may be active competitions and/or completed competitions. The list may be provided by any suitable means. Just as a few non-limiting examples, the list may be communicated by one or more of a web page, email, RSS feed, text message, and so on. The list may include such information as the name of the competition, a planned competition start date, and a reference to the workspace. The reference to the workspace may include a URL and/or other location indicia.
Each competition typically is for the development of one or more assets. Each listing may include a reference to one of a number of workspaces, typically a workspace allocated to that respective asset. The workspace may be any suitable workspace. Just as a few examples, not intended to be limiting, the workspace may be or may include one or more of a shared document, a wiki page, a web page, a blog page, a forum, a bulletin board, and so on. The workspace may be specific to the asset under development. In some cases, the asset to be developed is a part of a larger or combined asset to be developed at least in part by a series of development competitions, and the workspace may include documents and information associated with the larger asset.
The workspace includes a place for providing a draft competition specification 102. The draft specification may in some cases be first provided in a substantially complete form that would be used as the specification if no comments are received or as a rough draft that is expected to be further revised by the administrator, potential competitors and/or contest sponsors. One benefit of providing a draft specification is that potential competitors can review the draft specification and provide substantive feedback on the specification prior to the competition. The potential competitors can provide input that allows for the development of a specification that already addresses potential concerns of competitors. In a drafting process, for example, in which an architect or designer provides a draft, and a reviewer provides feedback to the architect or designer, this process of draft and revisions may be visible to those who have access to the workspace. Access to the specification drafting and revision process may give competitors further insight as to the desired asset to be developed in the competition.
In some embodiments, depending on competition and/or administrative policies, potential competitors may need to perform certain actions or have certain privileges in order to have access to the workspace and/or provide feedback. For example, competitors may be required to have participated in a number of competitions, have a rating in a type of competition, have a rating over a predetermined threshold, etc. in order to have access and/or the ability to provide certain types of feedback. Potential competitors may be required to have submitted contractual documentation, such as a non-disclosure agreement and/or an intellectual property assignment document or other documentation or certifications as a prerequisite to participation.
Feedback may be received on the draft competition specification 103 in the workspace. Feedback may be received from potential competitors, administrators, reviewers, competitions sponsors (e.g., customers) and/or other parties. Different parties may provide the same types or different types of feedback. For example, some participants, such as reviewers, administrators, sponsors, or potential competitors with particular prerequisites may have privileges to edit text, or edit certain sections, while others may only be able to mark text or provide comments. For example, if prize money and/or start date/times are provided in the competition specification, potential competitors may be allowed to comment on the logistics of the competition but not modify the competition specification. Potential competitors may be able to edit technical information in the specification or add technical information or comments to the specification.
Typically, a period of time prior to the competition is specified for feedback. For example, if the draft specification is first provided 2 weeks prior to a competition, a feedback period may be specified that is from two weeks prior to the competition to one week prior to the competition. The feedback may be any type of feedback. Just as a few examples not intended to be limiting, feedback may include any one or more of comments on the specification, suggestions for changes to the specification, text edits to the specification, suggestions for changes to graphics or designs, edits to graphics or designs, additions to the specification, general or specific comments on the specification and/or the competition logistics, and so on.
In some embodiments, the feedback may include an indicia of interest or disinterest in participating in the competition. For example, in some implementations there may be one or more “voting” buttons that allow a potential competitor to indicate interest in participating in that competition. In some implementations, indicia of interest may include a single voting button in which a potential competitor can indicate interest in participating in that competition. In some implementations, indicia of interest may include a two voting buttons in which a potential competitor can indicate interest in participating in that competition, or indicate that the potential competitor is not interested in participating in the competition. In some implementations, indicia of interest includes a degree of interest. In one such implementation, as an example, five radio buttons, each representing a different degree of interest are provided, in which a first button indicates that a potential competitor will participate, a second button indicates that the potential competitor is likely to participate, a third button indicates that the potential competitor may or may not participate, a forth button indicates that the potential competitor likely will not participate, and the fifth button indicates that the potential competitor will not participate. A comment area may also be provided for the potential competitor to give a reason for the indicated interest indicia.
In some implementations, indicia of interest may be recorded without storing information about who has indicated interest. In other implementations, the indicia of interest may be stored with information about the identity and/or qualifications of the potential competitor(s) who have provided a positive (e.g., will compete or are likely to compete) indicia of interest. Such information may be used to determine the likelihood of success of the competition. For example, in some cases if two or more potential competitors with ratings over a predetermined threshold (e.g., a rating threshold of 900, 1200, 1500, or 2200) have provided a positive indicia of interest, it is likely that the competition will be successful. Likewise, if, after a predetermined time period, there have been no indicia of interest by a predetermined number of potential competitors with sufficiently high ratings, it may be beneficial to take steps to increase participation, such as to increase prize money, modify timelines and/or logistics, and so on. Likewise, if there is an amount of interest above the predetermined threshold, it may be possible to start the competition early, or to reduce prize money or points, or to take other steps with some confidence that the competition will complete successfully.
The indicia of interest may in some implementations be publicly available. In such cases, all visitors can gain some understanding about the feedback with respect to interest expressed. The published indicia may include all information and/or may include a summary of the indicia. For example, the indicia may include a table demonstrating interest for range of skill level (e.g., totals of indications from potential competitors in the 900-1199 skill rating range, totals of indications from potential competitors in the 1200-1499 skill rating range, totals of indications from potential competitors in the 1500-2199 skill rating range, totals of indications from potential competitors in the 2200+ skill rating range, etc.)
The draft competition specification may be finalized based on the feedback 104. In some cases, this may involve simply “locking” the specification workspace to further edits. In some cases, this may involve a final review and edit of the specification by a designated administrator, review board member, and/or contest sponsor. In some cases, this may involve changes to competition logistics. In some cases, there may be one or more rounds of edits and comment periods.
In some cases a specification is not considered finalized until a number of potential competitors have provided an indication of interest in participating in the competition. For example, the start of the competition may be postponed to a date/time later than a scheduled time if sufficient indicia of interest are not received. In some cases, a competition may be scheduled to begin at some time period after sufficient indicia of interest are received. Techniques may be used to determine the likelihood of success based on expected participation, such as described in co-pending U.S. Provisional Patent Application Ser. No. 61/020,703 by Fairfax et al, entitled SYSTEM AND METHOD FOR CONDUCTING COMPETITIONS, filed on Jan. 11, 2008, the teachings of which are incorporated herein by reference.
In general, in another aspect, a system 200 for developing an asset by competition may include a competition list including a list of competitions, where each competition is for the development of an asset. Each element of the list may include a reference to one of a number of workspaces, where the workspace is relevant and/or allocated to the asset competition. As shown in the figure, Competition 1 includes a reference to Workspace 1, Competition 2 includes a reference to Workspace 2. Competition n includes a reference to Workspace n. In should be understood that there may be any number or combination of competitions and workspaces, and that the number shown in the figure is intended to be exemplary.
The system includes one or more workspaces 203 a, 203 b, 203 n, generally 203, each including a facility (which may include or communicate with a data store) for a competition specification 204 a, 204 b, 204 n, generally 204. The specification facility 204 may or may not have a provided at the time that a workspace is first created. At some time prior to the asset development competition, a competition specification 204 is provided in the workspace 203. In some embodiments, the workspace 203 includes facility for receiving feedback on the draft competition specification. The feedback may be received from potential competitors, administrators, competition sponsors, and so on. In some cases, the specification is designated as a draft for a period of time prior to the competition. At some time prior to or and the start of the competition, the specification 204, is designated as final. There may be an indication in the competition list of the status of the specification (e.g., not yet provided, available for feedback, feedback period closed, competition underway, etc.)
Competitions 205 a, 205 b, 205 n, generally 205, are held for the development of the specified assets using the specification 204 taking into account the feedback from potential competitors.
Referring to FIG. 3, in one embodiment, one possible generalized implementation of a competition for the development of an asset is shown. It should be understood that this implementation is but one of many possible examples of competitions for which the method of FIG. 1 and the system FIG. 2 may be applied, and that other types of competitions also may be used. The asset may be any sort or type of asset that may be developed by an individual or group. As non-limiting illustrative examples related to software, an asset may be a software program, logo, graphic design, specification, requirements document, wireframe, static prototype, working prototype, architecture design, component design, implemented component, assembled or partially-assembled application, testing plan, test cases, test code, documentation, and so on. As non-limiting examples unrelated to software, an asset may be a computer hardware or electronics design, or other designs such as architecture, construction, or landscape design. Other non-limiting examples written documents and content such as documentation and articles for papers or periodicals (whether on-line or on paper), research papers, scripts, multimedia content (including without limitation video, audio, graphics images, cartoons, sounds, graphical presentations, business presentations, etc.), legal documents, and more.
In some embodiments, the development process is monitored and managed by a facilitator 1000. The facilitator 1000 can be any individual, group, or entity capable of performing the functions described here. In some cases, the facilitator 1000 can be selected from the distributed community of contestants based on, for example, achieving exemplary scores on previous submissions, or achieving a high ranking in a competition. In other cases, the facilitator 1000 may be appointed or supplied by an entity requesting the development, and thus the entity requesting the competition oversees the competition.
The facilitator 1000 has a specification 1010 for an asset to be developed by competition. In general, a specification 1010 is intended to have sufficient information to allow contestants to generate the desired asset. In some embodiments, the specification has been made available to potential competitors for review and feedback as described with reference to FIG. 1 and FIG. 2. In some cases, the specification 1010 may include a short list of requirements. In some cases the specification may include the result of a previous competition, such as a design, wireframe, prototype, and so forth. In some cases, the specification may be the result of a previous competition along with a description of requested changes or additions to the asset. The facilitator 1000 may review the specification 1010, and format or otherwise modify it to conform to standards and/or to a development methodology. The facilitator 1000 may in some cases reject the specification for failure to meet designated standards. The facilitator 1000 may mandate that another competition should take place to change the specification 1010 so that it can be used in this competition. The facilitator 1000 may itself interact with the entity requesting the competition for further detail or information.
The facilitator specifies rules for the competition. The rules may include the start and end time of the competition, and the awards(s) to be offered to the winner(s) of the competition, and the criteria for judging the competition. There may be prerequisites for registration for participation in the competition. In some cases, the specification may be assigned a difficulty level, or a similar indication of how difficult the facilitator, entity, or other evaluator of the specification, believes it will be to produce the asset according to the specification. In some cases, in addition to the technical requirements, competition rules and logistics may be included in (or even if separate, considered part of) the competition specification.
The specification is distributed to one or more developers 1004, 1004′, 1004″ (generally, 1004), who may be members, for example, of a distributed community of asset developers. In one non-limiting example, the developers 1004 are unrelated to each other. For example, the developers may have no common employer, may be geographically dispersed throughout the world, and in some cases have not previously interacted with each other. As members of a community, however, the developers 1004 may have participated in one or more competitions, and/or have had previously submitted assets subject to reviews. This approach opens the competition to a large pool of qualified developers.
In some embodiments, the communication of the specification can occur over a communications network using such media as email, instant message, text message, mobile telephone call, a posting on a web page accessible by a web browser, through a news group, facsimile, or any other suitable communication. In some embodiments, the communication of the specification can be accompanied by an indication of the rules including without limitation the prize, payment, or other recognition that is available to the contestants that submit specified assets. In some cases, the amount and/or type of payment may change over time, or as the number of participants increases or decreases, or both. In some cases submitters may be rewarded with different amounts, for example a larger reward for the best submission, and a smaller reward for second place. The number of contestants receiving an award can be based on, for example, the number of contestants participating in the competition and/or other criteria.
The recipients 1004 of the specification can be selected in various ways. In some embodiments, members of the community may have expressed interest in participating in a particular type of asset development competition, whereas in some cases individuals are selected based on previous performances in competitions, prior projects, and/or based on other methods of measuring programming skill of a software developer. For example, the members of the community may have been rated according to their performance in a previous competition and the ratings may be used to determine which programmers are eligible to receive notification of a new specification or respond to a notification. The community members may have taken other steps to qualify for particular competitions, for example, executed a non-disclosure agreement, provided evidence of citizenship, submitted to a background check, and so forth. Recipients may need to register for a competition in order to gain access to a finalized competition specification.
In one embodiment, a facilitator 1000 moderates a collaborative discussion forum among the various participants to answer questions and/or to facilitate development by the contestants. The collaborative forum can include such participants as facilitators, developers, customers, prospective customers, and/or others interested in the development of certain assets. In one embodiment, the collaboration forum is an online forum where participants can post ideas, questions, suggestions, or other information. In some embodiments, only a subset of the members can access and/or post to the forum, for example, participants in a particular competition or on a particular team.
Upon receipt of the specification 1010, one or more of the developers 1004 each develop assets to submit (shown as 1012, 1012′ and 1012″) in accordance with the specification 1010. The development of the assets can be accomplished using any suitable development tools or system, depending, for example, on the contest rules and requirements, the type of asset, and the facilities provided. For example, there may be specified tools and/or formats that should be used.
Once a developer 1004 is satisfied that her asset meets the specified requirements, she submits her submission, for example via a communications server, email, upload, facsimile, mail, or other suitable method.
To determine which asset will be used as the winning asset as a result of the contest, a review process 1014 may be used. A review can take place in any number of ways. In some cases, the facilitator 1000 can engage one or more members of the community and/or the facilitator and/or the entity requesting the asset. In some embodiments, the review process includes one or more developers acting as a review board to review submissions from the developers 1004. A review board preferably has a small number of (e.g., less than ten) members, for example, three members, but can be any number. Generally, the review board is formed for only one or a small number of related contests, for example three contests. Review boards, in some embodiments, could be formed for an extended time, but changes in staffing also can help maintain quality. In some embodiments, where unbiased peer review is useful, the review board members are unrelated (other than their membership in the community), and conduct their reviews independently. In some embodiments, reviewers are allocated such that they only infrequently work on the same contests.
In some embodiments, one member of the review board member is selected as a primary review board member. In some cases, a facilitator 1000 acts as the primary review board member. The primary review board member may be responsible for coordination and management of the activities of the board. In some embodiments, the customer and/or facilitator 1000 serves as the only review board member(s).
In some embodiments, a screener, who may be a primary review board member, a facilitator, or someone else, screens 1016 the submissions before they are reviewed by the (other) members of the review board. In some embodiments, the screening process includes scoring the submissions based on the degree to which they meet formal requirements outlined in the specification (e.g., format and elements submitted). In some embodiments, scores are documented using a scorecard, which may be a document, spreadsheet, online form, database, or other documentation. The screener may, for example, verify that the identities of the developers 1004 cannot be discerned from their submissions, to maintain the anonymity of the developers 1004 during review. A screening review 1016 may determine whether the required elements of the submission are included (e.g., all required files are present, and the proper headings in specified documents). The screening review can also determine that these elements appear complete.
In some embodiments, the screening 1016 includes a preliminary substantive review and selection/elimination by the entity that requested the competition. For example, if the competition is for a wireframe, the entity may select the wireframes that seem to be the best. This smaller group may then go on to the next step.
In some embodiments, the screener indicates that one or more submissions have passed the initial screening process and the reviewers are notified. The reviewers then evaluate the submissions in greater detail. In preferred embodiments, the review board scores the submissions 1018 according to the rules of the competition, documenting the scores using a scorecard. The scorecard can be any form, including a document, spreadsheet, online form, database, or other electronic interface or document. There may be any number of scorecards used by the reviewers, depending on the asset and the manner in which it is to be reviewed.
In some embodiments, the scores and reviews from the review board are aggregated into a final review and score. In some embodiments, the aggregation can include compiling information contained in one or more documents. Such aggregation can be performed by a review board member, or in one exemplary embodiment, the aggregation is performed using a computer-based aggregation system. In some embodiments, the facilitator 1000 or a designated review board member resolves discrepancies or disagreements among the members of the review board.
In one embodiment, the submission with the highest combined score is selected as the winning asset 1020. The winning asset may be used for implementation, production, or for review and input and/or specification for another competition. A prize, payment and/or recognition is given to the winning developer.
In some embodiments, in addition to reviewing the submissions, the review board may identify useful modifications to the submission that should be included in the asset prior to final completion. The review board documents the additional changes, and communicates this information to the developer 1004 who submitted the asset. In one embodiment, the primary review board member aggregates the comments from the review board. The developer 1004 can update the asset and resubmit it for review by the review board. This process can repeat until the primary review board member believes the submission has met all the necessary requirements. In some embodiments, the review board may withhold payment of the prize until all requested changes are complete.
In some embodiments, a portion of the payment to a developer of one asset in a series of assets is withheld until the until after other competitions that make use of the asset are complete. If any problems with the asset are identified in the further competitions, these are provided to the reviewer(s) and the developer, so that appropriate changes can be made by the developer 1004.
There can also be prizes, payments, and/or recognition for the developers of the submissions other than first place submissions. For example, the contestants that submit the second and/or third best submissions may also receive payment, which in some cases may be less than that of the winning contestant. Additional prizes may be awarded for ongoing participation and/or reliability. Payments also may be made for creative use of technology, submitting a unique feature, or other such submissions. In some embodiments, the software developers can appeal the score assigned to their design, program, or other submissions.
It should be understood that the development contest model may be applied to different portions of work that are required for the development of an overall asset. A series of development contests is particularly applicable to assets in which the development may be divided into stages or portions. It can be beneficial in many cases to size the assets developed in a single competition such that work may be completed in several hours or a few days. The less work required to develop a submission, the lower the risk for the contestants that they will not win.
Referring to FIG. 4, in one illustrative example of a series of competitions, steps for development of a software application are shown. While described as an illustrative example in some places as a web application, it should be understood that any sort of software application, with any type of architecture, including without limitation mobile applications, client/server applications, thin-client applications, to give a few examples, may be suitable for this type of development process.
A series of contests may be held to develop the software application. For example, a person, referred to as a customer, may have an idea or concept for a software application. The idea may be, for example, thought-out in detail or only with a high level of description. A series of development competitions (with each such competition, for example, operating as shown with respect to FIG. 3) may be held, starting with the specification of the concept. For example, a contest may be held for the development of application requirements 1101. In such a contest, the initial description and documentation provided by the customer may be used as part of the contest specification 1010 (FIG. 3). The asset to be developed in the competition is the application requirements documentation. It may that the contestants need to interact with the customer to develop the requirements. Typically, this interaction would take place on a forum that is open to all of the contestants. The review of the requirements may involve one or more peer reviewers (i.e., members of the contestant community), as well as the customer. The selection of a requirements document may be based on the degree to which it represents the concept presented, the actual desires of the customer, and technical understanding and feasibility.
The requirements documentation that is developed in the requirements contest 1101 may then be used as part of the contest specification 1010 (FIG. 3) for a wireframe competition 1102. The wireframe contest 1102 may be held for the development of a wireframe (e.g., a visual guide used to suggest layout and placement of fundamental design elements in the interface design) of the application. The wireframe typically lays out the interface of the application, and presents visually the way that a user will interact with the application software. The review of the wireframes may involve one or more peer reviewers (i.e., members of the contestant community), as well as the customer. The selection of a wireframe may be based on the degree to which it implements the requirements, the actual desires of the customer, and technical understanding and feasibility.
When the wireframe contest 1102 is complete, a contest may be held for the development of a static prototype (e.g., an implementation of a web site in HTML or other markup language, typically without data persistence or other server-based functionality) using the wireframes as a starting point. The static prototype shows screen displays as they would look in the application, but does not have implemented functionality. The review of the static prototype may involve one or more peer reviewers (i.e., members of the contestant community), as well as the customer. The selection of a static prototype may be based on the degree to which it implements the requirements, the actual desires of the customer, and technical understanding and feasibility.
When the static prototype contest 1103 is complete, a contest may be held for the development of working prototypes (e.g., working implementations) based on the static prototypes. The working prototype is code that implements the requirements, wireframes, and static prototypes, along with any other comments or instructions provided by the customer. This working prototype may have certain restrictions or requirements that are described in the contest specification 1010 (FIG. 3). The working prototype may not need to be highly scalable, or enterprise quality, but merely good enough to try and permit customer use.
By having a customer involved in the requirements for and selection of the deliverables, a contest-based development process results in the efficient creation of software that the customer is happy with. At any stage in the process, if a customer is not happy with the final results, the customer can hold another contest to revise the previous asset with new or changed requirements.
The working prototype may be sufficient for some customers 1105 as a useful application. For others, the working prototype is a first step for confirming the desired requirements for a software application. Once they have used and tested the functionality of the working prototype, the working prototype may be used as the input to another series of competitions for development of an enterprise quality software application.
Shown as “STAGE 2” in the figure, another series of contests, beginning with the development of an application specification 1106 based on the working prototype may be held. In some such embodiments, a contest for development of an application specification 1106 may be held. The contest specification 1010 (FIG. 3) may include the winning working prototype and information about changes requested from the working prototype. Other information that may be included in the contest specification 1010 (FIG. 3) may include the required format and scope of the application specification. In one embodiment, the application specification is a requirements specification, including screen displays, functionality description, and so forth. A customer may participate on a review board for a specification, particularly to fill in any gaps, or to clarify any problems with the inputs to the specification competition. Of course, the specification competition could be held without a working prototype (if STAGE 1 is skipped) or just using wireframes and/or static prototypes as input. Likewise, a customer may just develop its own specification, and/or engage a consultant to develop a specification.
Once the winning application specification is selected, it may be used as the contest specification 1010 (FIG. 3) for an architecture competition. The asset(s) that may be developed as part of the architecture competition may include an overall architectural design, as well as a description of components that may be used to build the application. An architectural design may include a description of new components that may be built as part of other competitions 1108, 1109, or as existing components from a component library 1120. When a winning architecture is selected, the resulting component specification(s) may be used as input for component design competition(s) 1108. A customer may participate on a review board for an architecture competition, particularly if the customer has architecture expertise. The customer may be particularly interested in the interfaces and integration with its other systems. Typically, it is useful to have skilled architects working on the review board for an architecture competition, to identify technical issues with the architectural design.
In some embodiments, the components specified in the winning architecture may be developed by holding a series of component design competitions 1108. The winning component designs are then used as input for component development competitions 1109. As illustrative examples, component design and development competitions have been described, for example, in the above-referenced U.S. patent application Ser. No. 11/035,783. When all components have been completed, they may be assembled in assembly competitions 1110. Finally, test scripts may be developed, tested, and run in testing competitions 1111. At the completion of STAGE 2, an enterprise-ready software application that meets the requirements is completed.
In some embodiments, reusable assets may be provided to increase the speed of development in each of the various stages. For example, templates, graphics, tools, design patterns, and so forth may be used to increase the productivity of the contestants, and to give a common style to make evaluation and integration easier.
It should be understood that one, two, or more of the steps may be performed in a different order, or combined or omitted. For example, in STAGE 1, there may not be a need for a requirements competition 1101. Rather, the contestants in the wireframe may start from the description provided by the customer, and ask questions in a forum or otherwise to generate their wireframes, without use of more formal requirements documentation. Likewise, there may be at any stage multiple competitions and/or multiple levels of competitions. For example, for a complex application, there may be an overall architecture design, and then individual competitions for the architecture design of subassemblies. The architecture of the subassemblies may be designed, needed components built, and the subassemblies assembled. The subassemblies may then be assembled into complete application in an additional competition or competitions. Testing competitions may be held for various portions of an application, for example, for a subassembly, or for a distinct portion of an application, such as a user interface with another competition held for testing of back-end functionality.
It also should be understood that development by a series of competitions is flexible, in that contests can be repeated if the results were not as expected, or if additional changes or new functionality is desired. Likewise, a customer can undertake as much work as it likes through development by other methods, for example, by using internal staff or outside consultants. For example, rather than holding a contest 1104 for a working prototype, a customer can take the static prototype, and develop the working prototype itself.
In some embodiments, where the assets developed outside of the contest environment are being used as the input to another contest, it may be useful to engage reviewers to review the asset. For example, a static prototype review can be conducted on a static prototype developed by a customer before that static prototype is used as input to the working prototype contest 1104. As another example, in STAGE 2, an entity might develop a specification itself, engage reviewers to review the specification, make any desired changes, then use the specification in a contest for some or all of the architecture, hold a contest for development of some or all of the components, and then assemble the application itself. A review can be conducted on the assembly, and a testing competition held to develop test scripts and other functionality.
In some embodiments, a “project plan” competition may be held prior to the competitions described. In the project plan competition, contestants develop a plan for development of the customer's project needs. For example, the plan may specify a series of development contests and timeframes for developing the desired software application. The plan may provide approximate costs, based on historical or other data, for similar competitions. The plan may include development strategies, and assumptions about customer participation. A project plan may be specified as a series of competitions, which may then be monitored through interaction with a competition management server. One asset developed in such a plan may be a configuration of contests as specified in a project management system, such as that described in co-pending U.S. patent application Ser. No. 11/755,909.
In one embodiment, a competition web site provides a registration process for a development project. Following project registration, a customizable dashboard, or “cockpit” is configured with a variety of functional widgets with which a customer can begin the process of having development work done by a member community. The customer can launch competitions and track projects through delivery and member payment. In some embodiments, the web site project-specific public or private ‘group’ pages, generated by the customer from the cockpit, on which the nature of the project can be described, relevant attachments can be viewed, and the customer can openly communicate with members via a bulletin board or forum to answer questions, receive suggestions, and so on. In some embodiments, these pages allow participants to see the evolution of the project, interact, collaborate, provide status reports, and make delivery.
In some embodiments, representatives of a customer, and members in different roles, may have access to different capabilities for the project. For example, some customers may be able to post projects, while others may only be able to review status, and other review and approve payments.
In some embodiments, a dashboard page serves as an entry point for customers to interact with the TopCoder Community. From this dashboard page, a registered customer may launch and track projects, send and receive messages and monitor a variety of information sources. The dashboard may include a profile box with information about the individual, and a link to other information, or to update the information, a button that allows the launching of a project, a button or link for help and “how-to” information, navigation to other pages, such as contests and discussion forums, and in some cases space for broadcast messages or advertising, search field, and so forth. In some implementations, the page is customizable, with a framework allowing for “widgets” to be dragged and dropped into place into a user-configured order.
In some embodiments, the dashboard includes an interface that provides information about the user's projects, which may be updated dynamically, such as the project name, the status of the project, the name of the personnel associated with the project, the status of the project, the status of payments, the number of contest participants, the number of submissions, the project phase, whether there are new forum posts (and in some cases, information or a link to the post(s)).
In some embodiments, the dashboard includes a messaging interface that indicates whether new messages are waiting, and allows a user to compose new messages. In some embodiments, a team information interface shows information about individuals who are working on a current project, or, for example, a past project, or have been designated for another reason to be included. An RSS feed interface provides an indication of whether content has been added to a particular web page. A news interface provides information about new features, or, for example, newly-available components that may be used in a competition.
Competition List
Referring to FIG. 5, in an exemplary embodiments, an excerpt from a competition list includes such information as the date and/or date/time of the competition 501, the client for whom the competition is being run 502, the name of the project that the competition is associated with 503, type of competition and asset that is being developed 504, the technology category of the asset 506, the name of the asset 507, the status of the competition 508, and the personnel associated with the competition, such as the manager of the project 510, the architect and/or author of the competition specification 511, and a reviewer 512 tasked with reviewing the specification prior to contest launch. It should be understood that this excerpt is exemplary, and is intended to illustrate the types of information that may be found in such a list. Additional information may be included, such as the date the competition was first posted, whether or not there are competitor registrations associated with the competition, and so on. Likewise some information in the list may be excluded from all or some users. For example, some users may not have access to the name of the client 502, or the name of the project 503, or may be provided with a code name or number.
In some embodiments, the date 501 may include a date (as shown) or a more specific date/time indication of when the competition is scheduled to begin. The indication may be localized for the time zone of an individual. The client 502 listing may be the sponsor of the competition, and/or the ultimate recipient of the asset to be developed. The client listing 502 may be omitted for certain competitions, or for certain viewers who are not authorized to have client information. The project listing 503 provides an indication of which project the competition is associated. There may not be a particular project, or the project may be the name of a larger asset (e.g., a software application, a multi-media presentation) for which multiple or a series of asset development competitions are being held. The project listing 503 may be omitted for certain competitions, or for certain viewers who are not authorized to have project information. For example, in some cases, only administrators have access to the client and project information, while potential and actual competitors do not.
The asset type 504 provides an indication of the type of asset that is to be developed. Some of the types shown in the example include Graphic Design, Component Design, Component Development, and Assembly. Other asset types would be appropriate for other competitions. In some embodiments, the assets may include each of the assets described with reference to FIG. 4. The category of competition 506 provides an indication of the technology of the asset. This may be helpful for potential competitors to quickly identify competitions that may be interesting to them.
The name of the asset 507 gives a further indication of the technology and/or the larger asset that is being developed. For example, the Manifest Manager is shown as a .NET Generic component, and this competition is for the design of that component. Following design, there may be another competition for the coding of this component and such a competition may already be scheduled in anticipation of successful completion of this competition. Such a competition would have a similar listing (with a different date) and an indication that the type 504 is “Component Development.”
In this example, the name 507 of each asset is a link to a workspace for that asset. The workspace includes further information and details about the asset, as well as a web page allocated for the competition specification. In this example, the competition specification is provided in a “wiki” such that any user with sufficient permissions can make edits to the wiki, and add or delete text. The changes to the page made by a user can be viewed, so that a reviewer or administrator or another competitor can see who made what changes. In addition, there is a portion of the page dedicated to comments, such that a user can post comments, general or specific, about the technical description, competition logistics, or other specification information. Links to a “forum” is also provided, such that a conversation can take place among administrators (e.g., managers, architects), competitions sponsors and/or potential competitors about the competition.
The status 508 shown in this list is the status of the specification drafting, commenting, and review. The status of the specification can change from “Not Received” to “Comments” when a draft is made available for comments. Following the comment period, the status will change to “Final Review” which is when the designated reviewer will review the specification along with the designated architect and/or the designated manager. If review fails, the status becomes “Failed Review—Edits Required” until the specification passes review, and then it is “Complete.” Potential competitors thus can review the list, and review the specifications (if the potential competitor is qualified and has fulfilled any necessary prerequisites).
The names associated with the manager 510, architect 511 and reviewer 512 positions may be last names (as shown with the exemplary names listed), handles/pseudonyms, first names, and/or other identifiers. These names may be omitted for certain competitions, or for certain viewers who are not authorized to have full project information. For example, in some cases, only administrators have access to the names of the manager and architect information, while potential and actual competitors do not. In some cases, the names are provided, but they are “handles” (pseudonyms) used by members of a community of developers. In some cases, the handles displayed are designated for a role (e.g., Contest Administrator 231)
Workspace Implementation
Referring generally to FIG. 6, in some embodiments, a screen display provides an entry point for users to view competition status and to review and comment on a competition specification. It should be understood that the exemplary screen displays are not intended to be limiting, but rather are intended only to illustrate by example. The screen display may be implemented with HTML pages, with HTML/Javascript pages with AJAX display technology, with a client/server system, desktop application, browser plug-in, applet or in any other suitable manner. For example, in some embodiments, a commercially available content management system is configured to provide the described interface by accessing information from other web pages and/or back-end systems. In some embodiments, the screen display is integrated or presented with the contest system and other systems described in the referenced patent applications.
In some embodiments, the workspace is implemented using a customizable framework of widgets. In this context, the widgets are small functional components operating within a client software program, for example, a web browser, such that the client can present a number of these small applications as an overall combined application. The client may be another general-purpose or special-purpose application, however, preferably, the client is a web browser such as INTERNET EXPLORER from Microsoft Corporation, or FIREFOX from the Mozilla Foundation. Widgets may be implemented in any suitable technology, depending on purpose, context, and implementation, but in a preferred embodiment are implemented in HTML and Javascript. The widgets may operate as part of a client widget framework that is implemented in JavaScript. The framework utilizes AJAX communication with a data access layer on a server, which data access layer resides within a JBoss Application Server. Widgets may be implemented using the framework, in order to enable the described features.
In this example, the exemplary widgets include a “Competition Details” widget 602, which provides details about a competition. This includes some of the information from the list of FIG. 5, and may also provide additional information. A “Competition Status” widget 606, provides further details about the status of a competition, and may include links to further information about the competition status. A specification editing widget 608 displays the current draft of the specification, and allows for editing of the specification. This may be implemented as a wiki, such that changes to the text are tracked, and user permissions can be used to allow access. A discussion forum widget 610 provides a place for comments and discussion about the specification 608 and/or the competition more generally.
It should be understood that the figure shows a simplified screen display, and there typically would be additional features on the screen display, including without limitation additional information about this and other competitions, navigation aids (e.g., links, menus, etc.) to other places on the competition site, capability to get alerts to changes, for example by subscribing to RSS Feeds, and so on. It also should be understood that that the use of widgets is just one possible implementation, and any other suitable implementation for providing screen displays, for example, using a browser (e.g., HTML pages, Javascript, ASP, etc.) and using other approaches such as thin client, client/server, and so on. Display on mobile devices, television/entertainment video systems and another other suitable systems also may be used.

GENERAL APPLICABILITY

Although described here with reference to software, and useful when implemented with regard to software assets, the cooperatively developed asset can be any sort of tangible or intangible object that embodies intellectual property. As non-limiting examples, the techniques could be used for computer hardware and electronics designs, or other designs such as architecture, construction, or landscape design. Other non-limiting examples for which the techniques could be used include the development of all kinds of written documents and content such as documentation and articles for papers or periodicals (whether on-line or on paper), research papers, scripts, multimedia content (including without limitation video, audio, graphics images, cartoons, sounds, graphical presentations, business presentations, etc.), legal documents, and more.

Claims

1. A method for developing an asset by competition, comprising:

posting a list of competitions, each competition for the development of an asset, each element of the list including a reference to one of a number of workspaces, the one of the workspaces for a respective asset;

providing a draft competition specification in one of the workspaces;

facilitating feedback from potential competitors on the draft competition specification in the workspace for a period of time prior to the competition;

finalizing the draft competition specification using the feedback; and

holding a competition for the development of the asset.

2. The method of claim 1 wherein the workspace comprises a wiki.

3. The method of claim 2, wherein the feedback comprises changes to the specification.

4. The method of claim 1, wherein some of the competitions in the list are upcoming.

5. The method of claim 1, wherein some of the competitions in the list are active competitions.

6. The method of claim 1, wherein the asset comprises a software component design, and the specification comprises a requirements specification.

7. The method of claim 1 wherein the asset comprises an asset selected from the group of assets consisting of: software component, software component code, graphic design, architectural design, product design, product specification, article, creative work, essay, video, audio, multimedia work, and presentation;

and wherein the specification comprises a description of the desired asset.

8. The method of claim 1, wherein facilitating feedback comprises facilitating feedback from potential competitors and administrators.

9. The method of claim 1, wherein the list comprises a competition name, a planned competition start date, and the reference to the workspace.

10. The method of claim 1, wherein the reference to the workspace comprises a URL.

11. A system for developing an asset by competition, comprising:

a competition list comprising a list of competitions, each competition for the development of an asset, each element of the list including a reference to one of a number of workspaces, the one of the workspaces allocated to a respective asset;

a workspace comprising a draft competition specification and a facility for receiving feedback from potential competitors on the draft competition specification in the workspace for a period of time prior to the competition;

a competition system for holding a competition for the development of the asset using a specification that is based on the draft competition specification taking into account the feedback from potential competitors.

12. The system of claim 11 wherein the workspace comprises a wiki.

13. The system of claim 12, wherein the feedback comprises changes to the specification.

14. The system of claim 11, wherein some of the competitions in the list are upcoming.

15. The system of claim 11, wherein the list comprises active competitions and upcoming competitions.

16. The system of claim 11, wherein the asset is a software component design, and the specification comprises a requirements specification.

17. The system of claim 11 wherein the asset comprises a graphic design, and the specification comprises a description of the desired asset.

18. The system of claim 11, wherein the asset comprises a mechanical part design, and the specification comprises a description of the requirements for the desired asset.

19. The system of claim 11, wherein the list comprises a competition name, a planned competition start date, and the reference to the workspace.

20. The system of claim 11, wherein the reference to the workspace comprises a URL.