Can We Change the Speaker of the House

From CNS News:
“Boehner Won’t Rule Out ‘Balanced Budget Amendment’ That Permits Unlimited Federal Spending”

Is there any way, other than elections, to remove the “Speaker of the House”.
Not necessarily from office. Just that position.

Mr. Boehner is showing the country that he is not conservative and has no real desire to fix any problems.
He is demonstrating that he is a ‘Good old boy’, lifer, politician that is more concerned with keeping the status quot for the operations of the ever expanding U.S. Government over the needs of the country.

I am just wondering. Can we take Mr. Boehner and Mr. Kantor and put them both at the bottom of the list?