When I was a Human Resources Manager, my boss told me that one of the best ways to select a candidate for employment was to listen to your gut. He said, and he was usually right, that your instincts about a person were as important as their qualifications and how they responded to interview questions.
The listening to your gut theory works, as well, when it comes to accepting a job offer - or rejecting one. I once turned down a job, despite repeated phone calls offering me more money, because I didn't feel comfortable with the office staff or the company culture.
I spoke to someone the other day who withdrew her application for similar reasons. The office was old and stuffy, full of paper files, and it was secluded and not a pleasant work environment by any stretch of the imagination. She was lucky because she's employed and not in a position where she has to find a job for financial reasons. It was an easy decision for her to withdraw from consideration.
On a related note, if you're not selected for a job and you receive a candidate rejection letter it could be a good thing. If the hiring manager has decided you're not a good fit for the job or the company, they could be right. If that's the case, you probably don't want to work there anyway.
Rather than taking it personally, consider it an opportunity to move on to a job that is a better match for both you and the company.