The first Amendment:

"Congress shall make no law respecting an establishment of religion, or 
prohibiting the free exercise thereof; or abridging the freedom of 
speech, or of the press; or the right of the people peaceably to 
assemble, and to petition the Government for a redress of grievances."

This Amendment is absolutely fundamental to the framework of the 
Constitution. It provides for all other Amendments. It ensures the 
freedoms of the people and the states.

It is also NOT total.
There are conditions where the 'total' freedom of speech is rightfully 
abridged.
For example, you are not allowed to yell fire in a crowded building when 
there is no fire.
You are not allowed to burn a cross on your own lawn.

The control of the speech is by local, state and federal governments. 
For example, you are not allowed to sing the 'Star Spangled Banner' on 
the steps of the Lincoln Memorial, or any federally owned park.
In many states students are not allowed to pledge allegiance to the flag 
in public schools.
In some cities, you are not allowed to have SILENT prayer before a 
sports event.
So you see, the first amendment, as well as others, do have limits. 
There is precedent and, though all sides have their own complaints, in 
general 'We The People' adhere to these limits.

The mosque at Ground Zero is inflammatory. It will incite social and 
religious discord.
It is historical in that those planning to build it have done so each 
time they have conquered a town or country. It is not unlike planting a 
country flag in a new land.
Those that oppose this 'speech' are not dictating that NO mosque should 
be built, they are requesting that no mosque should be built THERE.

The builders of the mosque declare that it is to be a social awareness 
gesture of peace and understanding. To show tolerance to others of 
different faiths and ideas. And yet, despite great numbers of people 
opposing, they still intend to build, displaying the absolute lack of 
peace, understanding and tolerance.
The People have reached out and suggested other sites. NOT squelching 
any persons first amendment right, but suggesting that in the idea of 
friendship, understanding and tolerance, perhaps the better choice would 
be to move the site.

Still the answer is NO.

The First Amendment does not provide the 'absolute' right to expression 
and religion, and is to be handled with wisdom and care, just like all 
other rights bestowed upon a U.S. Citizen.
It is one of the Jewels of The Constitution of this great country, 
penned in indelible ink by men for men.
But is expected to invoked with wisdom, lest the meaning be lost in the 
noise of oppression, discontent and misuse.

ASKING the builders to move the site is proper.
Asking them, in the name of compassion and tolerance is acceptable and 
intended by the Constitution.