The rule of thumb for prop pitch vs RPM is a change of 150 to 200 RPM per inch of pitch, all other things being equal.. So you went from a 17 pitch prop to a 20 which should drop your RPMs by 600 RPMs. My personal experience with this is that this 200 RPM vs inch change usually only holds for about a 2 inch (typical increment in prop pitch between sizes) change. Beyond that , the non linear shape of the engine torques vs RPM curve shows itself and greater changes in RPM occur. Also, there may have been another reason that the original prop allowed the engine to hit the rev limit, most likely a slipping hub. If you still have access to the original prop, take it to a prop shop and have it checked out. While you are at it, ask them to check that prop's ACTUAL pitch and diameter. It is not uncommon for boat owners to have their prop pitch or diameter altered.