What does this mean if my boyfriend said this?

My boyfriend is confident, and a joker. He is a typical guy and loves making me laugh. He spoils me rotten and treats me so, so lovely. I was a virgin when we met (I was 16, he was 19). He had been in previous relationships and so had had sex with a few girls - something which didn't bother. We just spoke on the phone and we were talking about what we remember from high school and reminiscing. His ex girlfriend came into it and he was jokingly talking about how she got annoyed he would watch football all the time etc. We then got onto the discussion of relationships and how he spent �450 on her birthday presents, only for her to break up with him a week later after three years together. He would spoil her a lot and I can't help but feel he felt stronger about her than he does me... We have been together a year and although we went to high school together (we didn't know each other in school) he added me on Facebook a month after they split (June 09). We became very close in November 09. Yet on the phone today he was talking about how he got a lot of 'sexual' things while he was single, still joking. He then left the phone conversation with what I considered a horrible thing to say. He says we haven't argued in so long because he now doesnt get so jealous... Because he doesn't care. It's "my problem if I cheat" (I never have and never would). I thought he'd gotten more trusting but like he says, he doesn't care. If i were to cheat itd be my fault and my loss. When him and his ex split he cried a LOT and went on about how he didn't want to lose her. But now I feel like if He were to lose me, he wouldnt be phased. I have never worried about this before but I'm now not so sure he loves me like he loved her, despite always telling me he loved me so much more. I'm not like her, shes very girly and outgoing. I'm outgoing and still girly but I sit and watch football with him and prefer having a laugh than standing looking pretty. What do I do? Am I worrying for nothing?