LISTSERV 16.5 - CODE4LIB Archives

Code4Lib 2009 -- February 25, 2009

Draft notes: Breakout Session: Evaluating Open Source

Input welcome from attendees and anyone else.

----------------

This breakout session started from an informal discussion at a C4L
wine and cheese. The group brainstormed questions about questions to
ask for evaluating open source.

These questions are not absolutes and some of them (perhaps many of
them) will elude clearly objective answers. Also, again and again, the
group pointed out that the presence of a question did not translate to
a requirement or a judgment — these are assessment questions, many of
which will not be relevant to every project and will only translate to
meaningful criteria on a selective basis. Additionally, while many of
these same questions could be applied to any software, the consensus
appeared to be that it was helpful to ask these questions specifically
in the context of OSS.

1. “Openness” of open source

a. Describe the license(s) used

b. Is the code freely and publicly available? Is it easy to find?

2. Growth and growth management

a. How widely is the code used?

i. How many
organizations are known to use it operationally

ii. How many
times has it been downloaded

iii. Is usage
information tracked and reported?

b. How long has it been in use?

c. How many developers are actively involved in the project?

d. What is the commit activity?

i. How are
commits reported?

ii. Can commits
be tracked in real-time? How?

e. Describe the enhancement process.

i. Are
enhancement decisions publicly available? Who decides?

f. Describe bug-tracking: what tools, how bugs are evaluated and
prioritized

i. Is the
bug-tracking system publicly available?

g. Describe QA/testing processes.

h. How is the software updated?

i. Is there a migration path to the next version?

j. Describe the development planning model. Is there upgrade
planning? A commitment to a migration path?

k. What tools are provided for migrations and upgrades?

l. Has the project forked, and if so, describe

3. Community engagement…

a. Are there user groups? How large are they? How often do they
meet (f2f, virtually, etc.)?

b. Discussion groups, chat channels, etc.—presence, traffic, availability

c. Activity of support forum, length of support

d. Other characteristics of the software community: size, diversity

4. Governance

a. Describe the governance model (nonprofit, foundation, etc.)

b. Is the governance transparent? Describe.

5. Code and standards

a. Describe the architecture—languages, structure, etc.

b. Is the project using version control?

c. How available is the version control system?

d. Is there a commercial support option?

e. Interoperability—describe.

f. Error logging and reporting—describe

g. Scalability?

h. Security? Encryption?

i. Does it
provide security auditing tools?

i. How are permissions set and what are the default permissions?

j. What platforms does it run on, and how easy is it to
implement on each platform?

k. For dependencies, does it rely on current versions of those programs?

l. Does the code hew to de facto or de jure standards? Which ones?

m. Are key developers active in related standards work?

n. Does the code include proprietary-source codex, flash players,
etc.—and how is that handled?

6. Documentation

a. Is it complete?

b. Current?

c. Open?

d. Written to standards (e.g. Docbook or DITA)?

7. Innovation and quality

a. Is it cool at what it does? Is it useful? What’s its karma?
Does it work well? Does it solve a problem? that needs to be solved?

b. Is it easy to use?

c. Is it focused on end users (including librarians, if they are
the software’s end users)?

d. Ease of installation? Consistent results?

e. Accessibility?

f. Internationalization?

g. Business intelligence functions?

h. Incompatibilities?

i. Failures and deficiencies?

j. Awards, reviews, citations?

k. Certifications?