It is a symbol which has been abused by racist elements, and certainly some people use it for racist purposes while paying lip service to the historical aspects. However, the historical elements of the flag are real and undeniable and should not be ignored for the sake of a crass minority.
I went to a college which saw 10 of its cadets killed as a result of the Cadet Corps fighting as a unit at the Battle of New Market, and that legacy is a profound one where no other cadet unit has ever fought a battle as a military unit (unlike Europe, where it has happened numerous times). We had to fight to put the Confederate battle flag (not the Stars & Bars) on our class rings because certain class members thought it was racist. That galled me to no end because if you were at the school and learned the history, it was painfully obvious that the symbol on our ring was not racist any more than the Don't Tread On Me flag is anti-English. In any event, we got the flag, and others in the class could choose another design in its place.
So, in a nutshell, the historical importance of the Confederate flag, representing the war dead of the Confederate South, trumps the misuse of the symbol by racist elements for me.
Here's an analogy, though somewhat strained. I visited the Japanese war memorial shrine, Yasukuni, in Tokyo earlier this year. Enshrined within it are numerous class A war criminals. While visiting the shrine, I saw a Japanese veteran (not of WWII, he was too young though definitely upwards of 65) carrying the Rising Sun flag around, crowned with a sprig of cherry blossoms. Now, seeing as how that flag represents Japan's Imperial past and their horrific actions during WWII, I could have been offended by its display especially since as a US military member, my brothers in arms were killed in Japanese captivity with reckless abandon (think Bataan death march, among other events). Was I offended? Nope. The context of the display was not intended for offense but rather for remembrance and honor. I took it to heart in that context and appreciated this veteran's respect for his country's military traditions.
Compare that to Germany with the swastika. THAT is certainly a racist and hurtful symbol because it was the emblem of the Nazi party. The German cross, on the other hand, was the symbol of the Wehrmacht, which despite its association with Nazi Germany was a military fighting organization and NOT a genocidal movement.
Anyway, I'm rambling at 8 am on a Saturday!