How should the Karma system be designed?


#1

What factors should contribute to a users karma? The most important question is what are we trying to Measure? Authority? Trust? How well people write their opinions? How prolific of a contributor they are?


More options or more detail to start with?
#2

Personally, I want the system to be a measure of how much you have helped people. Here is a proposal:

You would receive points for adding various content with multipliers for some content types. For example:

  • Asked a question - 10 points
  • Asked a question that had X subscribers - 30 points (etc)
  • Added an option 20 points (no multiplier for votes as votes on options (for example “iphone”) have little to do with the actual content usually.
  • Added a Pro/Con - 8 points
  • Added a Pro/Con that received X votes - 20 points

However, I think we need another signal to both surface more signal to determine who is helpful and to help identify and reward awesome people…a “thank you” button. We should add the ability to “thank contributors” at the option level. Only people who at least added a pro/con should receive the “thanks”


#3

Agree with all your suggestions; for actual #of points per item it’s probably a case of watching how the scoring system affects the site and balancing the weightings to avoid broken scoring.

A few ideas, but no preferences:

  • Philosophy: a few core principles should govern these decisions:

  • Those who contribute something useful / helpful should be rewarded (for any positive participation; asking, answering, curating, promoting, etc).

  • Good naturedness should be rewarded.

  • Penalties should be avoided as these lead to destructive cycles (though blocking accounts for abuse / similar would be a valid exception to this).

  • Educating users should be encouraged - any mechanism to flag content as non-constructive / incorrect should include a low effort but polite way to inform the person what they should do to improve.

  • New users should not be treated as inferior (though safe-guards to ensure the inexperienced understand the site’s rules would be OK).

  • Scalable: any system employed should continue to work as the user base grows.

  • Anonymity of voting; if users are able to affect one another’s scores (e.g. upvote a user in some way) avoid any mechanism which could turn karma into personal popularity points (e.g. “Jon Skeet just answered a question; he’s always right; take my points!”).

  • Exponential Drag - some method to slow the growth of higher scoring users to decrease visible disparity.

  • Flare - a separate metric for those answers which give something more than facts. Whilst all answers should add value, having an incentive to make an answer more interesting, amusing, or poetic will help grow the community / give the site its own personality. Having this as a separate metric also keeps the regular scoring system a valid measure of contributed content. This can be based on user votes/likes.

  • Point cap - perhaps there should be a limit to what you can score for anything to avoid popular subjects / a popular link from HackerNews throwing the scoring out.

  • Custom score views; store raw data and allow user-contributed scoring mechanisms to be submitted / users to select which scoring system they use to view people/questions/answers. This is effectively A/B testing to the extreme; over time hopefully the systems will boil down to a handful which collectively work for most people.

  • Words over Icons - to avoid misuse, perhaps buttons saying was this useful, thanks, would be better than a generic + & 👍 (thumbs up) buttons.

  • Spartacus Defense - should any community moderator system be implemented allow users to rally the masses to their defense; if the community believes the moderators are being overly strict their ruling can be overthrown.


#4

I’m still not sure what should be done when it comes to votes, given the communal nature of answers. Coming up with a fair and effective way of partitioning those points may not be straight forward. For example, if you add an answer without content and someone else fills it in, most of the value came from another user.

Also we want to avoid rewarding people for adding the most popular answer, as that would encourage people to spend most of their effort on answers that are already well known and popular.


#5

Maybe multiplying by votes is a bad idea, at least for Options. I think it’s OK for Pros/Cons, but the same argument could be made for those too.


#6

This is a brilliant power user idea. Would effectively crowdsource the ideal ranking.


#7

@mijndert what do you think about the issue of using votes in the karma rankings?


#8

Good point.

Thinking out loud - I’ll refine soon but thought these thoughts may be of interest:

You’d also want to avoid easy ways of gaming the system; e.g. someone who creates one option with a nice overview followed by detailed and referenced supporting information should be rewarded more than someone who goes to a Wikipedia list of page and just adds every option listed.

I’ve tended to fall into the second category for the moment, which would make me someone attempting to game the points system; but I do intend to go back over those options to flesh them out soon…

Points per word wouldn’t work as a concise answer’s more valuable than a waffling one.

Maybe an impressions score - someone who’s consistently contributed to highly scored/followed items gets a higher score than someone who’s participated in a few high scored and some lower scored items. The user’s score is based on an aggregate of overall activity rather than x points per action. The effect of the contribution would have to take into account the date so you couldn’t just add a minor tweak to a popular answer for an easy boost.


#9

This system would very much reward the popular “Best android phone” style queries over the long tail "Best password reset library for node? style questions. I’m not sure that is necessarily a bad thing, but I would like to reward people to fill out content in the long tail as well. That said, any points system that scales with votes would have the same issue.

What about a system that uses a variety of signals to “qualify” a contribution. If I add an option and nobody edits it, or votes on it, or reads it, I get X points. But if I add an option and it gets votes, or someone clicks on the “get it here” link, or someone bothers to help fill it out we could assume it was a useful and valid contribution and give Y points. It wouldn’t continue to scale as more votes pile on, but it would use those signals to ensure it was a useful contribution.

So in general any contribution gives you points, but you get a once-off “bonus” if we deem it a “helpful” contribution.


#10

This topic is now pinned. It will appear at the top of its category until it is either unpinned by a moderator, or the Clear Pin button is pressed.


#11

Handling karma is a very tricky subject. It is the means to encourage contribution and to direct a style of contribution. I think losing karma is important, but that is even harder than deciding how to give it.

I don’t think points should be awarded to any action. Here are some examples:

  • Add an option and contribute two pros: 40pt
  • Add option with one pro, recieve X votes: 30pt
  • Add 10 options (anywhere) and add 30 pros/cons (anywhere), recieving at least X upvote (anywhere of the 10): 500pt
  • Ask question which gets two options: 20pt
  • Thank an option from your question: 10pt
  • Be thanked: 10pt
  • Ask question with X followers: Xpt

The principle here is that Slant will be a better place with a good collection of pros/cons and options. I liked the idea of giving thanks so I added that to the list. These are crafted to try and prevent “gaming the system.” Sure you can add hundreds of options/pros, but if you don’t capture the interest of one other person you’re doing something wrong, but if you do capture that interest then you should be greatly rewarded for your efforts.

However, I do think that votes people are giving your contribution should easily available, just don’t contribute to karma directly.


#12

Is this repeatable? Is this an additonal bonus on top of the other inputs?


#13

Repeatable, sure 20 and 60 another 500pt. I think it should be in addition to other bonuses that one may get for submission. There could probably other awards like 200pt for 30 pros which shouldn’t be a additive, so it likely isn’t going to be simple to add point senerios without consideration to other groups.


#14

Thanks for: Instead of just being able to say thanks, it might be useful to have a drop down menu so that you can specify why you are thanking them. For instance using images, video, or other not text media or maybe your answers are well written or well researched. These don’t necessarily need to be tied to points, but as I suggest in my next topic it can help sort users

Titles and Points: Karma points are good for a quick study of someone’s usefulness, but I would suggest (and perhaps this needs a topic all on its own) a system for Titles. For example if you provide Apple related answers, or add pros and cons to Apple related answers and you get a lot of votes for them you can get the title of ‘Apple Expert’. If you get a lot of ‘Thanks for: good writing’ you can get the ‘Well Spoken’ title. This way users can get a more in depth background of someone without having to delve too deep.

As for points I would suggest points for answering questions that have long gone unanswered. Otherwise those questions will just get pushed further and further back with little to no hope of getting answered.


#15

Another thought - have a karma budget per question.

  • Each question’s budget is determined by a value algorithm (e.g. number of views).
  • Instead of receiving quantified points for contributing, people take a percentage share of that question’s budget.
  • Each option, pro and con could equally be budgeted, a percentage of the question’s budget weighted by number of votes received.
  • Any contribution gets the same percentage within that area (e.g. someone who creates a pro gets the same percentage as someone who fills out the description for the pro, and the same as someone who corrects spelling errors in there). Whilst potentially unfair, it’s impossible to have a better system without intelligent intervention; since removing superfluous content or sparking an idea for content is potentially just as valuable as describing that idea in detail.
  • Contributions which don’t add value are rejected, meaning those contributors don’t take any karma percentage.
  • Given the budget is affected by the popularity / votes against that content, it’s in all contributors’ interests to make their answer popular / higher quality.

Still not a perfect system, but an alternate angle to approach the problem from which may yield interesting results.

Inspiration: http://www.slant.co/topics/1202/viewpoints/5/~what-is-the-best-karma-gamification-system-employed-on-a-large-community-site~hit-record-accord#3


#16

I don’t think we want to be tying points to popularity, fixing the spelling in a option that has no votes should get the same points as fixing one with 100.

Editing is a little tricking. I still don’t think it should be X points per edit. The main issue is identifying that the edits as valuable. The best I’m coming up with is for 10 edits Xpts, one of those edits will be placed in review and points are awarded if the edit is accepted.


#17

So here is what we think is a good & simple v1:

  • Level 1 Pro = TItle Only
  • Level 2 Pro = Title + Description
  • Level 3 Pro = Title + Description + Source
  • Added L1 Pro = 20
  • Added L2 Pro = 50
  • Added L3 Pro = 150
  • Added Option = 30
  • Added Source = 50
  • Added Description = 30
  • Edited anything = 10 (capped at 100 points/day)
  • Someone subscribed to your question = 20

We don’t think we can use votes as an input as there is pretty much no correlation to the quality of the user input. This system would highly encourage people submitting fully fleshed out Pros/Cons with sources. Thoughts?


#18

Seems ok to me. Not sure if starting at 10 is needed, but maybe it leaves some room for other scoring types.


#19

How about an ELO system supplemented with badges…

ELO would reward consistency and decrease separation between top and bottom users that is inherent in a completely vertical points system. A system like this would allow adjusting points distribution over time to change focus with less community friction.

Add to this a badge/flare/cosmetic system to reward things like consistency (add a “veteran” title beneath their name, or have their name in a different color), taking on new questions (give them a “desert warrior” badge), etc.

As a user it would go something like this. “Oh, one of the contributors name is just slightly orange instead of black like all the others, I wonder what that is all about.” Check the profile and see badges that say he’s helpful to newcommers, an expert in Python, thorough and explains things in a simple language. “Cool, so when this person talks about Python, I should listen”.

It gives contributors something to gain by being helpful and doesn’t create authority that’s infallible only because of insane amounts of karma points (see mrbabyman/karmanaut).

With this system you can get points relatively easy and understand what your strengths are as a contributor, but the real accomplishments come from being recognized by the community.


Btw, I’d avoid calling them badges. Better call them something like “known for”, “accomplishments”, etc. Less gamey sounding although that’s exactly what it is.


Also, this way contributors could be in charge of their domain, maintaining things that they are knowledgeable and updating them as they evolve instead of jumping around different subjects searching for points. It would reward deeper knowledge instead of wider overall knowledge.


#20

The issue with ELO (As far as I can gather) is the inputs need to be in the form of Wins/Losses as it’s a relative ranking system. Not sure we have the right inputs to do ELO?