Not Restarting

I have a 2006 115hp yamaha 4 stroke. I went out last week shut my motor off and would not restart. I waited a while and it restarts. When i get home i check my water seperator and changed my fuel filter on the back of the motor. I took it out today with the same results. I lifted the motor out of the water then dropped it back down. What do you know it restarts. I was told it maybe the VST filter. When the motor is running i have no issues and it runs great with no issues. This is puzzling to me i keep the filters changed with checking the water seperator constantly. Any sugggestions would be greatly appreciated.