Although there are parts of polyamory that are obviously sexual, such as people choosing to have sex in groups of three or more rather than the "usual" two, polyamory for me is about family, rather than sexuality or sexual behaviour. My family-of-choice are the people that I enjoy spending time with and choose to be around. The core of my family consists of my partners, my partners' partners, my gay best friend and his boyfriend. They are the people I love.
Now, my gay best friend and his boyfriend are fiercely monogamous. Neither of them are going to have capital-R relationships with anyone apart from each other. But they are very much part of my family unit. For me, being poly is about extending connections. So my gay best friend's boyfriend became part of my family because he is a loved one of someone I love, and over time, he became important to me for himself. Likewise, even though I hardly know my girlfriend's husband, he is part of my family through our shared partner.
Very few of the people in my family-of-choice are people that I'd ever be sexual with, even if I did have a normal sex drive. Yes, being poly means that I get to have sex with a cute boy and a cute girl and no one minds about that too much. But more important to me, being poly means that I bake cakes for my girlfriend's girlfriend, have a toy otter left on my sofa, cuddle two people when watching TV, and sit by myself with a book because my boyfriend and girlfriend (who aren't involved with each other) want to talk about stuff too geeky for me.
I suppose what I'm saying is that although polyamory is part of my sexual orientation, it's a much bigger part of my relationship orientation. And I spend a lot more time having relationships with people than I do having sex with people. Pretty much all of my non-family-of-birth relationships are strongly governed by principles that come from my polyamory. (And thinking about it, maybe that's why my family-of-birth relationships bother me so much).