The middleman exists for a reason. An open dictatorship of the oligarchy would be too unstable. The paradox is that open power would be against the interests of the very oligarchy.
And the representative system you have today, even if not deem democratic, still 'works'. With this system the elite have a way to work their internal strugles in a non-violent way.