Trust in relationship

Three weeks ago I discovered my partner of 23 years had been texting a work colleague secretly. I asked for details but he refused to tell me anything except that he had finished it a week before when she had asked to meet up.
I do believe him when he says nothing physical happened.
He says it started because I no longer pay him any attention and she made him feel good.
Anyway, he told me that it was over and he no longer had her number.
We agreed to try again and we had been getting on well, talking, texting, flowers, going out etc.
But last week I discovered he still had her number in his contact list under a different name. I confronted him and he said it was because he still wasn't sure what he wanted. I made him block and delete her number while I watched, he then got very moody and managed to make me feel guilty.
He has apologised for this and said that I did the right thing and that he still loves me.
I do still love him and want to try and rebuild our relationship, but can I ever trust him again, I just can't get rid of the feeling I'm being taken for an idiot.
Any help or advice is gratefully received.