Sometimes I feel like somebody trying to "nip" my fly very subtle
I'm not sure is it a 4" trout or big steelhead but sure it's a fish, not snag... and when I'm trying set the hook... nothing
What to do in such situation, maybe downsize a little bit?
Its very possible that thoes are trout but even if they aren't you should never set the hook. I have never had a steelhead that didn't set the hook for me. the fish will take the fly and return to it's lie before it spits the hook out and by that time you'll have him hooked.. The only thing you should ever have to do on a steelhead is tighten up on them. There are two exceptions.
1. when fishing a dead drift fly you'll need to set the hook when the line twitches or curves upstream.
2. when fishing a skated fly on a short line, in that case you'll need to drop your rod tip and not tighten up until you feel the fish.
All other fish will hook themselves. Any grabs you don't hook wouldn't hook if you set anyway so it's best to do nothing just cast and wait for a fish to be on.