Ohloh Tracks Open Source Developers

eldavojohn writes "The startup company Ohloh has a database listing 70,000 developers working on 11,000 open source projects. Their aim is to 'rank' open source developers, which raises some interesting questions about exactly how useful this tracking company is. Questions like, 'Is there an accurate way beyond word of mouth to measure the importance and skill of a developer?' I found it slightly alarming that, to this site, the number of commits (with input from the number of kudos) tells how good a developer you are."

Accurate? Not for me

[mrslacker (1122161)]

I don't know how representative it is, or if it might improve over time, but I looked myself up.

I found mentions in 5 projects - _except_ they're all just versions of 2.6 kernel source with the same contribution for an obscure TV card cx88 variant I did. In practice, I'm sure I'm hardly alone in having contributions (mostly in small ways, but sometimes very considerably) to over 100 projects over the years. I guess I have to go through and add some of those projects.

Naw, CBA. At least I can make sure my resume is accurate.

So What Metrics Do You Suggest?

[eldavojohn (898314)]

I'm sure I'm hardly alone in having contributions (mostly in small ways, but sometimes very considerably) to over 100 projects over the years.

I also don't think you're alone in finding that metrics fail to measure good programmers. My boss constantly asks me for lines of code count from developers. No matter how many times I express this to him, this is not a measure of success or of how good a coder you are.

I tried to think of metrics to relay up the chain (a special thank you to the stat-scm goal in maven) but I come up with some pretty lame ones:

• Code to comment ratio is desired at 1:1 (at least in the commercial world)
• A class/method/function/procedure/module desired size should be defined and rated
• # of Unit tests

As you can see these are the ones that I found could be automatically gathered. And even these have exceptions. Anything else I think of either takes too much time to gather or is subjective. This is tough, I would like to default to peer review but oftentimes I find teammates voicing their personal hatred for an individual or taking into account personal qualities when ranking a developer. Real Life Example: Teammate A is from MIT and teammate B thinks everyone from MIT is a god. Unfortunately Teammate A hasn't done anything but criticize everyone's code without any constructive comments to make it better.

I submitted this story hoping it would open dialog on measuring coding abilities in a semi-automated way.

Re:So What Metrics Do You Suggest?

[JonTurner (178845)]

# of penguin stuffed animals in your possession.
# of kernel builds
# of ICQ shouting matches ending in "Nazi!!!"
# of cans of Jolt consumed
# of steps from mom's basement to side door.

Or the old standby, lines of code (including comments, of course).

In short, there's no way to automatically judge the value of a programmer based on silly metrics. How would one score for "bugs not written" or "elegance of solution"/"nasty kludge avoided"?

FOSS is worth whatever the users and the coders say.

No, no no!

[morgan_greywolf (835522)]

of penguin stuffed animals in your possession.
# of kernel builds
# of ICQ shouting matches ending in "Nazi!!!"
# of cans of Jolt consumed
# of steps from mom's basement to side door.

No! You got it all wrong! Everybody knows that programmer productivity is inversely proportional to the number of Slashdot posts!

Re:

[ohlolo (837547)]

Software development metrics are not worthless. They are, however, seriously misunderstood. This is partly why we built Ohloh to focus on Open Source: it's the world's largest testbed of available software development metrics.

One challenge to interpreting development metrics is having a clue about what is 'normal'. Just knowing your FOOBAZ count is X doesn't help much. Once you can compare your FOOBAZ count to 100k other developers, it may begin to give you some helpful perspective. Of course, relying on

Re:So What Metrics Do You Suggest?

[The End Of Days (1243248)]

What about some slightly deeper tracking into the repository? Figure out how much code a developer committed that had to be changed later.

Re:So What Metrics Do You Suggest?

[morgan_greywolf (835522)]

What about some slightly deeper tracking into the repository? Figure out how much code a developer committed that had to be changed later.

Not bad, but it depends a lot on the maturity of the project. Many young projects start out with several complete rewrites before they find a workable solution that performs well, is scalable, robust and reliable. And even on mature projects -- a lot of times and entire codebase is scrapped to accommodate new features: think Samba 3.x -> 4.x or GNOME 1.4 -> GNOME 2.0.

But what's the point?

[EmbeddedJanitor (597831)]

Making OSS dev is hardly a competition sport, so why do this? Kudos is one thing, but that should come from the community, not from some database.

Some people will get a shiny glory and some will feel annoyed bbecause their projects/contributions have not been tracked.

Re:

[lymond01 (314120)]

It's a little tricky because talent is only as good as what's seen. From a business perspective, they don't care so much about the getting there, as long as there's something done within deadline (but they want to know you're working to meet that deadline so they count your lines of code).

Nice thing about open source code is that it is reviewed by other developers as it goes. This site might be worthwhile if it has good input from thoughtful people. Like Slashdot people! The shiny...happy kind...! Righ

Re:

[samkass (174571)]

The SEI and various other organizations focused on process have a lot of suggestions. There are some simple ones, such as "function points" * "complexity" as a measure of productivity. Or defects per function point as a measure of quality. Or some from the extreme programming world of user stories completed.

But nothing is really truly accurate, only guides humans. If it were, managers could be robots.

Re:

[timewasting (1230064)]

I always hated the lines of code metric because it was so useless. A really good architecture with a good coder leads to much LESS code that looks simple to the casual observer. Similarly, the number of comments might not be very useful as well. Spaghetti often has lots of extra comments, while good code is often self-commenting (at least in my commercial realm). Lots of unit tests are often unhelpful if there are no automated system tests. The best coders balance design, coding and testing in the scop

Re:

[eh2o (471262)]

Here are some more that can be derived automatically:

- Code cleanliness (e.g. consistent camelCase, whitespace, comment styles)
- Statistics of commits over the same section of code (may imply that it is buggy)
- Rate of API breakage -- how long does a function last before it's name/args/return value changes. This metric could be "weighted" by evaluating the prevalence of the function. A high rate implies an unstable design.

Some of the ones I don't like that Ohloh uses:

- "Mature" meaning that the project ha

Re:

[bishiraver (707931)]

Number of unit tests doesn't necessarily denote good unit tests.

It would be a better metric to show what percentage of your codebase is actually covered by unit tests.

Using Test-Driven Development ensures that this percentage is notably high.

sounds familiar

[by Anonymous Coward]

This reminds me of how academics are increasingly judged. It is more about how many papers and how many other people link to it rather than the quality of each paper's work or the note of the linking party. Accordingly, many authors inflate their 'impact' scores by splitting up papers and publishing nonadvancing science, no-one can blame them for this as many are trying to justify themselves to their departments or are still doing the postdoc merry-go-round looking for new jobs every 18months.

no thanks

[dcskier (1039688)]

You can't effectively rank developers. First there are just too many to rank. Even in college football, where thousands of people are paid everyday to monitor it, they don't try to rank all of the ~119 Div 1 teams, just the top 25. Secondly there isn't a simple metric to rank developers. It's about as smart as saying look I did the most work on this project because I wrote the most lines of code.

This could even have a negative effect if developers get concerned about their ranking and try to game t

Same here

[jaaron (551839)]

Most of my contributions were on website documentation, wikis, or mailing lists, which aren't included in these metrics. At the moment, a lot of my commits are done on repositories not directly available to the public. While I don't really need Ohloh to tell me if I've contributed to a project or not, it's still a little annoying.

And what about contributors who submitted patches that had to be committed by someone else? Or people who contribute by providing help on IRC channels, blogs, forums, or other mailing lists?

While ohloh metrics can be useful, they also need to be taken with a grain of salt, particularly the contributor metrics. They're a bit more useful on measuring a project as a whole (but they still miss a lot of activity).

Listed twice

[solprovider (628033)]

I am listed as two people with the same pseudonym; my real name is not found. I am listed for two related projects belonging to the same organization. Both of me have the same score albeit for different skills. Ohloh obviously only checks commits to the main branches; my commits and LOCs to an experimental branch of one project would drown my official commits. I won commit status due to my assistance on the mailing lists and a lengthy complicated patch for critical functionality; my name is in the credi

business model

[magarity (164372)]

What the heck is their business model, or is this just a hobby site? About the only way I can think of to make some money is to take some under the table in exchange for a higher rating.

Re:

[darthflo (1095225)]

TFA says classifieds, them digging through their data on request, subscriptions for them monitoring projects/devs.

I imagine they might make quite nice head hunter "equipment". Think company x wanting to incorporate SMB connectivity to their closed source product (and requiring a developer to do so). Instead of digging thru or placing classifieds, Ohloh would hook them up with the top 10 contributers to smbfs. This could turn out to be a great deal for both sides. Company X gets a dev really skilled in th

Their model is simple

[CarpetShark (865376)]

It's a very simple model really, when you think about it. Let's examine their possible train of thought:

Sites can sell advertising when they get lots of frequent users. Sites need users to get users. Sites need some kind of user list to bootstrap. Where can you get a big list of users from? Why, isn't that opensource stuff based on lots of people communicating in the open, over the net? Oh, hey, let's use those suckers. Hmm. How can we make more suckers sign up after the first ones? Hmm... we need

Would this discourage contributers to open source?

[softwaredoug (1075439)]

Would this discourage contributers to open source projects? Now if I put on my resume that I've contributed to an open source project, somebody is going to want to look me up. I have to deal with all that baggage when I just wanted something to do in my spare time. Also, I really am not sure I feel comfortable being given an absolute rank. People always bring different skill/approaches to different jobs and I don't think you can arguably say one is better than another. I've worked in teams where everyone respects the different capabilities and limitations of each member. Its sort of like arguing there is an absolute thing known as "intelligence". Is there really such a thing or do we just all bring different skills/perspectives/approaches to the problems we solve? I'd prefer to think the latter, that everyone contributes what they can but has their own limitations. Talking about absolute "intelligence" or "value" seems condescending and elitist.

Re:Would this discourage contributers to open sour

[Bananenrepublik (49759)]

Most open source projects' commits are already gathered on cia.navi.cx [cia.navi.cx]. I don't see what ohioh can add besides a link to the real name, which is easy enough to find out anyway.

This has been a problem at Wikipedia for a while

[daveo0331 (469843)]

http://en.wikipedia.org/wiki/Wikipedia:Editcountitis [wikipedia.org]

Number of commits?

[Tarlus (1000874)]

So in other words, I could commit some of my own code to a CVS repository, find some errors that I missed, fix them, commit it again, decide to add more comments, commit it again, find one more thing I probably could have done differently and then rewrite it, commit it again...

And I would be ranked highly as a great developer?

Re:Number of commits?

[rucs_hack (784150)]

Certainly as a diligent one who actually cares about code quality.

Anyone who thinks they can produce bug free code first try is an idiot. Anyone willing to accept that there are always going to be bugs, and actively looks for them is a good coder.

Re:

[Aladrin (926209)]

Or was too stupid to code it right the first time.

I'm not calling him stupid... That was the example he was trying to get across. By this metric, stupid programmers that commit a lot because of mistakes are rated as highly as highly-motivated, caring programmers who commit a lot because they have a lot of additions to make.

Re:

[rucs_hack (784150)]

What?

You can't be a very good coder yourself.

Making mistakes != stupid coder.

Making mistakes, finding them and fixing them makes a good coder, nothing else.

Sure the fix might not be great, it might be an ugly hack that needs to be refined, but you're only a bad coder if you don't fix your mistakes, not if you make them in the first place.

Re:

[Tanuki64 (989726)]

This is moderated 'insightful'? But definitely not by real coders. More by wannabes like a previous project manager of mine, who, whenever he found one of my bugs, complained I would not test my code. I just wonder how he knew that when he found one, that I previously did not remove 500, which I found myself through testing.

L337 development skills!

[hellfire (86129)]

So in other words, I could commit some of my own code to a CVS repository, find some errors that I missed, fix them, commit it again, decide to add more comments, commit it again, find one more thing I probably could have done differently and then rewrite it, commit it again...

Your willingness to fix errors, add comments, and do code rewrites puts you in the pantheon of programming gods! The next thing you are going to tell me you actually write your own legible "how to" user guides in PDF!

Re:

[bdaehlie (537484)]

I think each commit is weighted by the rating of the project you commit to. So no, that probably wouldn't get you very far.

A couple commits to Firefox, the #1 rated project, is probably worth many commits to a less highly rated project. I'll let you go to their site and figure out how they rate projects.

Pedantic tip...

[Otter (3800)]

"Kudos" is not plural, just a word that happens to end in "s", like "pathos". "Kudo", as used on that site, is as meaningless as "etho" or "mytho". The more frequent references to "many kudos" or other treatments of it as discontinuous are also incorrect, although much less jarring.

Flawed, but interesting.

[urcreepyneighbor (1171755)]

Wouldn't such a system assume that everyone uses only one handle - or, their real name - all the time for every project? If so, then a lot of people - who contribute under multiple handles, nicks, whatever you want to call their identities - are going to missed or severely under-rated.

I would rather not have my real name attached to most of what I've contributed. One, because my code is so damn sloppy that it's embarrassing. Two, because I don't want the hassle of my real life - you know, offline - and my, uh, "digital lives" conflicting with each other. Three, if I was easy to find - online - I run the risk of being pestered with silly tech support questions.

UrCreepyNeighbor, while an accurate description of my personality, is one of many identities I have. Same could be said of almost everyone. I'm sure "HotChic17CA" doesn't use that username when she's talking with her grandmother, for example.

Re:

[ohlolo (837547)]

Disclaimer: I co-founded Ohloh. We let people 'claim' development contributions individually. So your Ohloh account can be related to as many (or as few ;-) nicknames on projects as you like.

Re:

[Knara (9377)]

What stops me from claiming nicks which are not my own?

Re:

[grumbel (592662)]

Wouldn't such a system assume that everyone uses only one handle - or, their real name - all the time for every project?

You can register on the page and link all the different aliases back to together to refer to a single person.

I think its a much bigger issue that all those people sending patches will be ignored, since there isn't really a standard way in most version tracking systems to keep track of the patch submitter instead of those that actually commit it into the repository.

Slashdot

[sm62704 (957197)]

You only have to comment on slashdot to tell that this is a really bad idea. You have people modding a comment "troll" because they don't like a stated opinion, for example. You have people modding a first post as "redundant" and a spot-on comment as "offtopic". People suck, especially at judging other people.

And on a thing like that, you may have someone who knows absolutely nothing about code making judgements about coders.

It's a stupid idea. It actually sounds like some harebrained idea thought up by a P

Re:

[Tridus (79566)]

If the first post is "FIRST!", Redundant isn't bad. I mean thats been done how many zillions of times already? At this point it is redundant, but in the context of the site as a whole rather then one individual article.

Of course if they gave me "-1 (Idiot)", I'd use that on the "FIRST!" people instead.

Re:

[LMacG (118321)]

I don't think he was referring to the canonical "first post", but to the first posting of an idea/thought/comment, which is subsequently repeated by somebody else. The somebody else should get the redundant mod (given due diligence in checking timestamps and allowing leeway as necessary).

And I'll agree with the GP that this Ohloh thing sounds like it came straight from a PHB.

Re:

[sm62704 (957197)]

I won't argue with that, but I've seen first posts that were actually commenting about the topic itself that were modded "redundant". In fact I try to not submit a comment until I see at least one other posted, because otherwise it will be modded to oblivion and nobody will see it, so what's the point of making the effort?

Fortunately it seems that whenever I happen to hit slashdto right when they've posted something new I'm invited to metamoderate, so by the time I'm done with thet there are at least a doze

What An Incredibly Inane Idea

[Jane Q. Public (1010737)]

This is completely ridiculous. They are attempting to judge value when there simply is no objective measure for the kind of things they are trying to judge.

I think I would object to having my name listed on this site, even if the "rating" were high.

Re:

[Tanuki64 (989726)]

They are attempting to judge value when there simply is no objective measure for the kind of things they are trying to judge.

This is only half true. You can judge the quality of code. How good does is comply to the OO principles? Makes the design sense? Does it look maintainable? Robust? Reusable? How is it documented? There are even some metrics, which could be measured by static code analysis programs. Nevertheless I doubt that for each with Ohloh registered software project a senior software engineer

Sing for it!

[lymond01 (314120)]

Like the bards of olde, OS devs don't code for money. They code for prestige and fame amongst their fellows! Surely this site will decide who is the greatest dev to walk the earth. And that dev will have his own code set in stone and copied for ages to come. That developer will be legend.

Unless, heaven forbid, the voting is more like the U.S.'s political system.

Advogato

[NaCh0 (6124)]

It's been done before. It is called advogato [advogato.org]. This is a site where developers can join, blog, and rate each other based on a trust matrix.

Other frustrations

[djmitche (536135)]

I work on Amanda, but the site misrepresents my contributions in two important ways, too: first, I commit a lot of other peoples' patches, so my name appears in the ChangeLog a lot less often than it appears in the commit log. Second, Amanda changed from CVS to Subversion a few years back, and Ohloh doesn't index the old CVS submissions. As a result, the project is marked as just a few years old (it was originally written in '92), and few of the many historical contributors are not listed. I would like t

Quantitative metrics dont work on developers

[KillerCow (213458)]

Quantitative metrics don't work on developers. As soon as a developer learns what it is, they are smart enough to game the system.

I [commit] can [commit] game [commit] any [commit] system [commit] based [commit] on [commit] commit [commit] counts[commit].[commit]

Numver off bugz fixd es eze 2 gaeme two.

Bug free code and low bug recidivism is easy. [have tester check code before checkin].

Number of projects? Sure. Every possible sub-component now has it's own source tree and project space.

Lines of code? Su

Open source playing cards to help funding!!!

[syousef (465911)]

I can just see the tv ad:
"Wow! Collect yours today"
Then 2 kids in school uniforms
"I'll trade you my RMS for your Linus and Eric S Raymond!"

Nonsense

[Jane Q. Public (1010737)]

It's not a measure at all. For example, on some projects a ajor contributor might not even have commit rights... the code would have to be committed by others. And so on.

They are attempting to measure something for which there is no consistent measure. As a consequence, there is no question that their "ratings" MUST be distortions.

Re:commits

[krog (25663)]

It's as good a measure as any.

You might be right, but it still sucks. And in the case when all your solutions are crap, I think it's dishonest to present any of them as actual, workable solutions.

Re:

[kelnos (564113)]

Agreed. And note that Oholoh appears to, by default, only track 'trunk' branches for subversion repositories. So if I spend 6 months and 500 commits working on something on a branch, which I later merge into trunk (one commit), that history isn't tracked. Sure, you can add the branches/ dir to the Ohloh 'enlistment' (what a stupid word), but it seems to be at least frowned upon.

And not all commits are code. A decent percentage of the commits in my projects are i18n/l10n-related. Those are even harder

Re:

[epine (68316)]

It's as good a measure as any.

Talk about damning with faint praise. That's how I aspire to be evaluated: lined up naked against a wall while my vital statistics are transcribed by a group of bonobo monkeys. Hey, it's as good a measure as any.

In fact, the monkey-measure is probably better than commit-count, because no matter how my spam box bulges, the monkey-measure is less likely to persuade me to exchange an effective work habit for an ineffective work habit in an effort to sway a useless statistic.

People who fail to vomit when lin