I am thinking about removing more people too - if you never or hardly ever comment on my posts, but you do read them, comment in reply to this one so I know you are still here and don't take you off the list :) People who are reasonably good friends of mine in real life don't need to bother - it's only online-only aquaintances I'm likely to remove.
I've also set up a "non-default view" friends group for when I have more time and/or are bored with my work and want more journals to read, but don't want to view the unfiltered list in its entirety, with the spammy communities and all.