Matt -- it's not your imagination! That's exactly the type of behaviour I've seen in the pond where I've had my best success.
After I caught or hooked a few fish on the easily accessible side of the pond (over the course of several trips) I found that I couldn't catch a fish on that side of the pond no matter how carefully I stalked them or presented the fly. It probably didn't help that a few of the fish were sporting new lip ornaments. Out of frustration, I decided to try the other side of the pond even though the shore is extremely muddy and the water is only inches deep for several feet out from the shore. I hooked 4 fish that day! Of course, I had to walk ankle deep in the malodorous goo in order to land and release the fish (hence the mud bath) and I almost lost my sneakers more than once. I'll try the muddy side again under the guise of scientific research, but this time I'll wear my scuba boots or maybe even my waders!