Ohloh Tracks Open Source Developers

eldavojohn writes "The startup company Ohloh has a database listing 70,000 developers working on 11,000 open source projects. Their aim is to 'rank' open source developers, which raises some interesting questions about exactly how useful this tracking company is. Questions like, 'Is there an accurate way beyond word of mouth to measure the importance and skill of a developer?' I found it slightly alarming that, to this site, the number of commits (with input from the number of kudos) tells how good a developer you are."

Accurate? Not for me

[mrslacker]

I don't know how representative it is, or if it might improve over time, but I looked myself up.

I found mentions in 5 projects - _except_ they're all just versions of 2.6 kernel source with the same contribution for an obscure TV card cx88 variant I did. In practice, I'm sure I'm hardly alone in having contributions (mostly in small ways, but sometimes very considerably) to over 100 projects over the years. I guess I have to go through and add some of those projects.

Naw, CBA. At least I can make sure my resume is accurate.

So What Metrics Do You Suggest?

[eldavojohn]

I'm sure I'm hardly alone in having contributions (mostly in small ways, but sometimes very considerably) to over 100 projects over the years.

I also don't think you're alone in finding that metrics fail to measure good programmers. My boss constantly asks me for lines of code count from developers. No matter how many times I express this to him, this is not a measure of success or of how good a coder you are.

I tried to think of metrics to relay up the chain (a special thank you to the stat-scm goal in maven) but I come up with some pretty lame ones:

• Code to comment ratio is desired at 1:1 (at least in the commercial world)
• A class/method/function/procedure/module desired size should be defined and rated
• # of Unit tests

As you can see these are the ones that I found could be automatically gathered. And even these have exceptions. Anything else I think of either takes too much time to gather or is subjective. This is tough, I would like to default to peer review but oftentimes I find teammates voicing their personal hatred for an individual or taking into account personal qualities when ranking a developer. Real Life Example: Teammate A is from MIT and teammate B thinks everyone from MIT is a god. Unfortunately Teammate A hasn't done anything but criticize everyone's code without any constructive comments to make it better.

I submitted this story hoping it would open dialog on measuring coding abilities in a semi-automated way.

no thanks

[dcskier]

You can't effectively rank developers. First there are just too many to rank. Even in college football, where thousands of people are paid everyday to monitor it, they don't try to rank all of the ~119 Div 1 teams, just the top 25. Secondly there isn't a simple metric to rank developers. It's about as smart as saying look I did the most work on this project because I wrote the most lines of code.

This could even have a negative effect if developers get concerned about their ranking and try to game the system instead of making quality contributions to projects.

business model

[magarity]

What the heck is their business model, or is this just a hobby site? About the only way I can think of to make some money is to take some under the table in exchange for a higher rating.

This has been a problem at Wikipedia for a while

[daveo0331]

http://en.wikipedia.org/wiki/Wikipedia:Editcountitis [wikipedia.org]

Number of commits?

[Tarlus]

So in other words, I could commit some of my own code to a CVS repository, find some errors that I missed, fix them, commit it again, decide to add more comments, commit it again, find one more thing I probably could have done differently and then rewrite it, commit it again...

And I would be ranked highly as a great developer?

Re:commits

[krog]

It's as good a measure as any.

You might be right, but it still sucks. And in the case when all your solutions are crap, I think it's dishonest to present any of them as actual, workable solutions.

Re:business model

[Anonymous Coward]

Their website says they have independent investors. Maybe this is just a technology demonstration, or maybe they'll sell subscriptions when it's working well, or maybe they'll allow people to buy homepages to better present their skills, or allow companies to buy a customized search for programmers they want to hire. I can think of a dozen possible ways to make money off this.

Remember, a startup virtually never ends up doing what they start out doing. (The Apple guys started out making phone-hacking blue-boxes.) Even if they intend to make it big by tracking open-source developers, I doubt that will be their primary business in 5 years.

It's kind of sad that the an initial response to an interesting new company is 'they're not profitable today, so the only possible business model is bribery'.

I'm in the same boat: working at a startup with private investment but not taking money from users yet. We've been accused of being a front for various giant .coms (I wish!). Trust us, when we have something that's good enough that users will want to pay for, we'll have no problem flipping the switch and charging users for it. Until then, just chill.

Listed twice

[solprovider]

I am listed as two people with the same pseudonym; my real name is not found. I am listed for two related projects belonging to the same organization. Both of me have the same score albeit for different skills. Ohloh obviously only checks commits to the main branches; my commits and LOCs to an experimental branch of one project would drown my official commits. I won commit status due to my assistance on the mailing lists and a lengthy complicated patch for critical functionality; my name is in the credits, but the commit is credited to someone else. I cured a critical bug in a third project (a component affecting many FOSS projects), but my patch was included as text in the bug tracking system and committed by and credited to someone else. The system measures lines of code, but great programming often reduces LOCs (as happened in the patch to the third project.) I am about to post code in a bug-tracking system that will not be integrated into the project due to management's objections ("This function has been broken for 8 years and we are afraid to fix it"); this will not be credited. Contributions to official documentation are ignored, as is useful information on personal websites.

Another committer has a much higher score. He is involved in more products and has committed tons of code to the official branches. He has kudos from other developers. From researching several developers whose work I know, I agree with the scores (except mine. My fame is in the corporate world for completing critical, urgent IT projects. I am a very minor player in the FOSS world. I am surprised my score is so high based on the limited contributions considered.)

Some suggestions:
1. List one person as one person. The organization requires a unique pseudonym for all projects. Start with the page translating the pseudonyms to real names.
2. Look at mailing list activity and distinguish between questions and answers. The first post in a user thread is a question; additional posts are often requests for information; the final post not from the original poster is usually the solution. The first post in a dev thread is often a new idea; additional posts are clarifications so give credit for length to avoid "+1" posts but avoid crediting lengthy log listings.
3. Look at bug-tracking systems. Many official commits are patches provided by non-committers through the bug-tracking system.
4. Counting LOCs is usually a poor determination of the quality of contributions. One of my official commits added code. My commit was reverted and replaced with half the LOCs of the original code. While I would accept credit for the concept that was implemented, those LOCs should not contribute positively to my score. Did the committer writing the excellent replacement code lose points for reducing LOC?

Ohloh ignores many important types of contributions and poorly assigns credit for much work to the committer rather than the contributor. Ohloh does not distinguish between types of commits; many commits are correcting bugs introduced by the same committer. Assigning importance ratings to commits requires integration with bug-tracking systems; CVS and SVN do not have scoring mechanisms.

Ohloh cannot score personality. Much of my career has been accomplishing the "impossible" (whether functional or due to deadlines) as my focus is on business usefulness rather than technical limitations. One person (scored 9 by Ohloh) is an incredible programmer once someone proves something can and should be done, but is extremely resistant to new ideas. Creativity is not reflected in the scores.

Ohloh has a good idea. I like the kudos system (and hope merit wins over popularity.) The system still needs work to be accepted as a reliable source of useful information.