How do I deal with someone I thought was my friend telling people we are sleeping together?

My husband passed away a little over a year ago. My daughter just turned 4 and I just turned 40. So we're not dealing with school-age bullying, we are adults. I have a male friend I've known since middle school and is recently divorced, He moved back to our hometown after being gone for years. He knew my husband in school and also of his sudden death. We started hanging out together often, just talking about his divorce, what I was going through, and how both of these things were affecting his son and my daughter. It wasn't anything I wouldn't do with a female friend. I really thought of him as a great friend and he always seemed so compassionate to my feelings when I was talking about my husband. He really was ultimately concerned for my well-being and my daughters'. He started making little comments as time went by that made me wonder if I was giving him the wrong impression. Or some sort of flirting that may be misunderstood, I didn't know. We had the "talk" (several times-over about 6 months time). He always seemed so understanding, and told me he knows we are just friends, and he's not taking it that way, and I'm going through alot, and I have my daughter to be worried about right now.....and etc.....always the "friend". Will be there if we need anything...blah, blah, blah..Come to find out...He's been leaving this open for people to draw the conclusion that we are, and have been having sex when we are together. I am beside myself with anger and humiliation. I could only imagine how people think of me as a "grieving wife" and mother. I honestly don't know how to handle this...please help..my heart feels ripped out, again.