“drive someone crazy” = make someone go crazy
“My younger sister drives me crazy with her terrible jokes.”

“What are you driving at?” = What do you mean?
“So, why did you say that terrible stuff about my clothes? What are you driving at? Do you mean they are old and ugly?“