I'd be more concerned about this little shakeup in the White House with Bannon were it not for the fact that historically there has been "weird shit" to quote @Walter_Sobchak quoting Dubya throughout the history of White House.
Andrew Jackson is famously quoted regretting that he didn't shoot Henry Clay or hang John C. Calhoun.
And of course Vice President Aaron Burr blew away Alexander Hamilton in a duel before being accused of treason.
Or how Woodrow Wilson was basically incapacitated after suffering a stroke for the final year of his presidency and his wife Edith essentially acted as the Executive.
Or the run-of-the-mill marital infidelities of Warren G. Harding, or John F. Kennedy or Bubba Clinton or - as it has come to light - George H.W. Bush aka "David Coppafeel".
The biggest issue of electing a President who is not a politician and a political outsider is the fact that Donald Trump didn't have the Roladex of political contacts that a normal candidate like a Jeb Bush or Romney or Hillary Clinton has. If Donald Trump wanted to build a golf course in Wales or a Hotel in New Jersey, he has 40 years of contacts whom he could lean on to get the job done. But when it came to the election, Trump had to pretty much rely on the leavings what with the likes of JEB! or Rubio or Cruz gobbling up all the available political talent. Add to the fact that Trump's political campaign was very lean in terms of money spent, he didn't have too many quality foot soldiers he could plug into the White House when he got there.