In fairness to the breakout session -- and to the issue of measurement in
general -- the point of the session wasn't to create some sort of magic
omnibus checklist (though the conversation kept drifting in that direction
and I kept pulling it back). In fact, if you look at the notes for the
breakout session, they are very reasonable questions to ask about any
software, open source or otherwise. Even if you look at one software program
and no other, they are still very sensible questions to ask, and I wish
people asked them more often. The metrics on that list certainly didn't come
out of a vacuum.

Having said that, I agree with Joe about his reservations with scorecards in
general, and I agree with Jonathan about associating the loose entity of
"Code4Lib" with an award that broad. I also worry about an award in the
context of such a small development community.

I could see something like "new app with the most potential for Z" or "most
interesting new floomajabbie for resolving the issue of X."

-- 
-- 
| Karen G. Schneider
| Community Librarian
| Equinox Software Inc. "The Evergreen Experts"
| Toll-free: 1.877.Open.ILS (1.877.673.6457) x712
| [log in to unmask]
| Web: http://www.esilibrary.com
| Be a part of the Evergreen International Conference, May 20-22, 2009!
| http://www.solinet.net/evergreen


On Wed, Mar 11, 2009 at 6:39 PM, Joe Atzberger <[log in to unmask]> wrote:

> I appreciate Jonathan sounding out the arguments against the proposed form
> of the award, and offering some alternatives.  In short, I think I agree
> with him.
>
> I was at Karen's "OSS Metrics" breakout session, and had a lot of
> reservations about the "output" of the session, even though the discussion
> there was interesting and well-intentioned.  It comes down to the two
> decision-making processes: the internal c4l one for making the award and
> the
> external one(s) being influenced by it.
>
> We were listing criteria one might use to evaluate a given project.  And it
> was a good enough list of issues, but I kept thinking that it was bound to
> fail if it were a scorecard to be used *comparatively* between otherwise
> heterogeneous projects on different platforms, in different environments,
> with different purposes, etc.  I wasn't even confident of our ability to
> review one individual criteria like "security" for a given project, let
> alone amongst all projects.  For the amount of work and expertise it would
> take to evaluate that honestly, we could be contributing *fixes* to even
> the
> "lesser" projects.
>
> But I'll put aside the question of how accurately we could pick amongst
> totally diverse projects.  Pretend we could.  I don't think we could
> communicate the objective context to the external decision makers who would
> consider themselves informed by the mere fact of the award.
>
> The Journal featuring a project has none of these problems, because it can
> maintain context.  Like "Is this project useful to archivists in major
> institutions?" or "Is this OSS project a good alternative to a different
> proprietary software X?"
>
> I also like the role of code4lib being more of a contributer and less of an
> arbiter.  If the goal is to benefit the cool projects, keep the money, show
> me the code.
>
> --Joe Atzberger,
> LibLime
>
> On Wed, Mar 11, 2009 at 12:46 PM, Jonathan Rochkind <[log in to unmask]
> >wrote:
>
> > As I think about the award idea more, I still don't really like it.
> (Sorry
> > Eric!).
> >
> > Some comments at
> >
> http://bibwild.wordpress.com/2009/03/09/why-i-dont-like-the-code4lib-code-award-idea/
> >
> > With a shorter version below (thanks Jodi).
> >
> > The award will inevitably be seen as an endorsement of the awarded
> project
> > by ‘Code4Lib.’ While some supporters say this is not the intention, I’ve
> > also seen supporters say the reason they want the Code4Lib name on it is
> so
> > the award will have more prestige. To me, this implies that an implied
> > endorsement in fact is part of the idea: What else would this prestige be
> > for? But whether it’s intentional or not, it’s inevitable.
> >
> > The Code4Lib community has indeed garnered a fair amount of prestige
> > lately, including by people who don’t really understand the informal and
> > non-official nature of Code4Lib. I’ve seen Code4Lib erroneously referred
> to
> > as an ‘organization’ several times. Much of this audience will see such
> an
> > award as an endorsement of the project awarded, by the prestigious
> > ‘Code4Lib’.
> >
> > But I don’t think Code4Lib actually has the capacity to accurately and
> > useful determine value of an open source project.
> >
> > Libraries need to learn how to evaluate open source projects on their
> own,
> > for their own circumstances and needs. Libraries, always on the look-out
> for
> > shortcuts, are going to be really tempted to use a Code4Lib award as a
> > shortcut to their own investigation. If it’s awarded by Code4Lib, it must
> be
> > good. I worry about anything that discourages libraries from the hard
> work
> > of developing their own capacity to evaluate projects; and I also worry
> > about such an implied endorsement actually steering them wrong because I
> > don’t think we have the capacity to reliably make such universally
> > applicable evaluations as a community. Sure, the award won’t be intended
> as
> > such, but it will be read as such.
> >
> > I would actually love to see a regular “notable project review” feature
> in
> > the Code4Lib Journal, perhaps in every issue. This could cover only
> articles
> > that the reviewers thought were exceptionally good, or it could cover any
> > project of note.
> >
> > And reviews would have particular reviewer’s bylines attached, making it
> > clear who was doing the evaluation, and discouraging the reader from
> > thinking it’s the “Code4Lib community”, which isn’t capable of speaking
> with
> > one voice anyway (nor do we desire it to).
> >
> > If the goal of the idea is to inject some money into library-domain open
> > source software development, than rather than an award with compenstaion,
> I
> > think the money could more effectively be spent funding an internship or
> > some kind.
> >
> > Perhaps something like Google Summer of Code. Give a stipend to some
> > library student (or currently un- or under-employed Code4Libber, but I
> like
> > the idea of getting library students involved as bonus) to work on a
> > Code4Lib community project. Perhaps the community could vote on which
> > project(s) were eligible for such an internship, and then people could
> apply
> > expressing their interests, and a smaller committee would actually match
> an
> > intern with a project.
> >
>