 |
 |
On Tue, Aug 9, 2011 at 3:50 PM, stuart yeates <[log in to unmask]>wrote: > ... > Ohloh is great. However it relies almost completely on metrics which are > easily gamed by the technically competent. Use of these kinds of metrics in > ways which encouraging gaming will only be productive in the short term, > perhaps the very short term. > > For example: it's easy to set up dummy version control accounts and there > can be good technical reasons for doing so. It's easy to set up a build/test > suite to update a file in the version control after it's daily run and there > can be good technical reasons for doing so. But doing these things can also > transform a very-low activity single user project into a high-activity dual > user project, in the eyes of ohloh. > > Turning on template-derived comments in the next big migration handles the > "is the code commented?" metric. > > The more metrics are used, the more motivation there is to use tools (which > admittedly have other motivations) which make a project look good. > > I agree the ohloh metrics are easily gamed. What metrics do you recommend that can't be gamed but still provide a synopsis of the project for evaluation, comparison, and selection? I think there is some utility even though they can be gamed. The metrics are not a substitute for critical evaluation, but provide a nice synopsis as a jumping off point. For example, if I am interested in projects that have a demonstrable lifespan > 5 years, and that have had more than 10 developers contribute, I can find that via these metrics. I can then assess for myself if any of the resulting projects are false positives (e.g., the commit log will give some idea of the types of commits made by each person). If you're concerned about the system being gamed via metrics, then you should also be concerned about user-submitted project descriptions. Projects have a tendency to over-generalize on what their software does, under-report defects, and generally paint a rosy picture. Will there be some sort of quality control/editing/verification of the claims made by submitters? Will it matter if some of the projects are described more generously than in reality? Won't the system still be useful even if they are? Matt