Freedom of Speech Conundrum

Following from the Charlottesville clashes, Freedom of Speech is once more in the news. This raises the question of whether Freedom of Speech, which is highly valued in western democracy, is absolute.

Is it valid to grant the Freedom of Speech as a defence to those who would deny Freedom of Speech to others if they came to power via democratic means - or indeed violent means?

As with most things there are two sides to every story.

1. Some would say it is right to grant freedom of Speech in all cases, as the only way to expose the inconsistencies and dangers in extremist ideologies is to counter them with public argument. The problem here is that this depends on there actually being a public debate when the silent majority are known for their silence and the extremists don't engage in debate.
2. Others would say that withdrawing the right to Freedom of Speech from certain organisations, such as fascists or communists, would allow authoritarian governments to arbitrarily declare government opponents as being censored, which is not good in a supposedly free society and could possibly itself lead to a totalitarian state. Who gets to pin the label of fascist or communist on the supposed fascist or communist?  Fascists have a tendency to label all their opponents as communists and communists have a habit of labelling all their opponents as fascists. The spectres of McCarthyism and Stalinism come to mind.
3. Some forms of Freedom of Speech are indeed illegal already, and rightly so, such as falsely shouting; "Fire," in a crowded theatre, or hate speech, which can be incredibly hard to define in some cases.

Answers on a postcard below.

What I find strange about the Charlottesville affair is how Trump is very quick to condemn Democrats for the most petty of reasons, yet takes days to condemn Neo-Nazis who come to a demonstration tooled up with semi-automatic weapons and brandishing swastikas. The Minute Men come to mind.

Trump gives new meaning to the phrase Trumped Up Charges.