my heart is confused and i don't know what to do

My ex boyfriend moved back to town, it's been 3 years since our breakup. Our relationship didn't end on good terms, he had to move to a different city because of college and long distance didn't work out for us. In the end, he ended things with me to be with a different girl. That was my first heartbreak and it tore me apart when he left me. I really thought that we would be together forever. Moving on forward to present day, I now have boyfriend and we have been together for about 6 months. Sadly, this summer he had to move because of college. We've decided that we are gonna make it work. Let me tell you this boy is the perfect guy out there so when he said that he had to move there wasn't a part of me that was scared of him leaving me for another girl because I trusted him and I 100% know that he is not that kind of guy. Long distance has been hard but up to this point everything is okay until my ex moved back into town. We have mutual friends so our friend group decided to hang out for the weekend, this is the first time I have seen him for years after our break up. I know what I'm feeling is wrong but when I saw him again, all my feelings for him came rushing back. He talked to me like before, we joked around and it felt so nice being around him again. After that weekend trip I realized that I wasn't really over him after all these years. But at the same time I don't wanna leave my current boyfriend, I love him I do but why do I catch myself daydreaming that my ex would come back to me. I think I'm going crazy I haven't told anyone about this and it's eating me up.