Do Americans think less importantly on someone who doesn't speak perfect English?

You don't have an accent and you sound better than some Americans! You do however need to learn more words. I think this person's beef with you is that you don't express yourself well. You hesitate and say meaningless things until you can find the right words to say what it is you want to say. Americans are impatient. You need to get to the point!