“there’s only like four people that post on a regular basis”

Then ban or hide posts on a case by case(mods) basis. You rarely purge stuff, so just send the down voted post to the sub-domain of shame. Have the comments post with a single word of “keep” or “purge”. Use a script to determine if the number of “purges” determine that the post deserves to go to the shame locker (something difficult to get like a 90/10 or 80/20 of “purge” to ALL comments).
Have anything that hits a “keep” of 90 or 80% bumped to first post for say 24 hours (after say 5 comments to reduce false calls). Have a search of posts that qualified for “keep” or “purge” for additional shaming.

You get user generated ban-hammer, and then you get the popcorn out to watch everybody troll the rating system. Everybody wins!