I don't have a strong opinion on whether that'd work, but it sounds a lot like the Slashdot comment moderation system. Perhaps someone with a good level of experience of posting on Slashdot can help shed some light on how it works out in practice.
I think it works well (been reading and posting to /. for about 15 years), and if I designed a forum then it'd work the same way. The trick is it forces you to justify your opinion in some way. There isn't a "-1 Disagree" or "+1 Right On" so to mod a comment you have to find some actual flaw with it, or justify why you think it should be raised above the rest. If you pick a dumb reason that clearly can't be justified based on the text, meta-moderators will wash things out later.
That said, it's not perfect. There are -1 Overrated and +1 Underrated mods. I would not have these if I was king for a day, because:
• They apparently somehow bypass meta-moderation, or have done in the past (!) so not surprisingly they get abused a lot
• They are vague and don't force you to justify what's good or bad about a post, in many cases I have seen abuse of these mod types devolve into reddit style agree/disagree modding
• They don't adjust the adjective that goes with the post, so for a long time and maybe still now you can get posts like +5 Troll or -1 Insightful, which is just confusing and generally bogus
Stripping out Underrated and Overrated would improve things a lot. I'd maybe make +1 Funny count for karma too because funny posts deliver a lot of value to the slashdot reading experience, IMO, and I see no problem with encouraging them.
I think the balance of adjectives is important though. There is no -1 Wrong or -1 Misinformed mod and that's important. If someone posts something polite, earnest and completely wrong there's no justifiable word to mod them down with. Instead you have to wait for a reply to correct them and then mod that up as Informative, Insightful etc.