This procedure is from the trawler forum that is emailed to me each day, I copy the good tips into notepad and file them, this may help:
------------------
Here are my thoughts. I've found that no matter whether I use a roller or a brush, getting the gel coat to a smooth shiny finish requires sanding, repeated applications and more sanding, until, after sequentially reducing the grit, you've gotten to the point where rubbing compound is used and then wax.
That is if you want a perfectly smooth, mirror finish... a worthy goal, but one that then requires constant attention to not using the boat in a manner that may ding it. For us, that's not a very reasonable goal, so we don't even try.
For all glass work that I do, I brush it on- resin or gel coat. The bigger the job, the bigger the brush. (I just have found that rollers aren't as effective and they are a constant source of hassle, more expensive too.) However, there are two qualities of disposable brushes available- cheap brushes that shed bristles like a shaggy dog sheds hair, and less cheap brushes that shed less. If you use the less cheap brushes and you'll find yourself picking out bristles less often.
After the gel coat is applied or maybe, if I haven't put in wax (sanding aid), two coats, then I'll give it a sanding, fill in any imperfections with more gel coat (or thickened gel coat) and re-sand. Keep doing this until it is as good as you want. Each coat is sanded until the substrate is reached, in other words, I'm using the gel-coat to fill and level, that way the gel-coat doesn't get thick except in the low areas. This can be done with resin, but gel-coat sands easier, though is more expensive.
I keep this up until I get the feeling that the next coat of gel coat can be the final coat. Then I sand carefully enough to not sand through, sequentially reducing grit until I get to the level of (im)perfection that I'm looking for.
If I do sand through in a spot, I just re-coat it and re sand, being more careful to not sand through and to feather the edges. If you are kinda new at this, don't get frustrated, just keep re-coating the spot and re-sanding, you'll get the hang of it within a couple of re-coats. On the other hand, if you keep sanding through in spite of being careful, that spot is probably too high anyways, so sand it down, then re-coat with gel coat.
For the jobs that I do for other folks, they get a mirror finish, but then they are paying for that level. For our use, I just get rid of the tree leafs and as much of the brush marks as I feel like and put on a coat or two of wax (most of the time). It's the wax that keeps the gel coat from chalking, not its thickness.
Much of the repeated gel-coating, sanding and re-gel coating can be minimized by filling as many of the imperfections as possible with resin, even thickened resin if need be, but there will probably always remain pin holes that will need to be filled, which can be filled with gel coat (If I need to I use cab-o-sil/silica to thicken white gel coat because it blends in.).
The objective of your design was to make a boat that is as light as possible, so as soon as you switched to fiberglass, that objective is no longer a consideration. In my view, now switching to low maintenance and having a finish that will hold up under heavy use is a more reasonable goal. Trying to keep the weight down by making everything, such as the thickness of the gel coat, as thin as possible maybe, once you switched to fiberglass, a poor placement of effort. Don't want to over do it, but don't want to try for the lightest fiberglass boat that is possible, but finding yourself constantly fixing cracks because it was on layer of glass to little.
However, your concern over the gel-coat being too thick and cracking is a valid concern, but unless you make it excessively thick, or if your design is too flexible, I doubt that you'll get a thickness that will crack... once ready for the final gel coat, one or two brushed on layers should be adequate but, at the same time, they will probably not too thick.
One thing to keep in mind, and to my way of thinking it is a good thing, is that no matter how badly you screw it up or how badly the gel-coat gets scratched or cracked, all you need to do to fix it is to sand, apply more gel-coat and sand it down to blend it in... that is if the underlying lay-ups are adequate. So make sure the boat is glassed well enough (thick enough) to not flex and your finish should be okay. If not, and you want to fix it, whether you do it now or years from now, sand it, apply another layer of glass and gel-coat it.
Just keep one thing in mind, it's only a boat. If it keeps the water out and you in, it's a successful boat. Now, the fact that you're constantly itching... as the saying goes- "been there, done that and better you than me". (grin)