I know that I see many fishermen wearing waders in their kayaks as they travel from point A to point B. Personally, I would never do this - even while wearing a PFD.
The way I see it - even if you can swim with your waders on it will surely be much easier to get back into the yak without the waders. When I paddle I usually wear fleece pants and a thin fleece top (not loose fitting) or just a bathing suit and a fleece top. The fleece offers protection against hypothermia (a serious killer when you turn over a yak - even in the middle of summer when in the ocean waters of New England). The extra struggling with the waders on will only result in greater heat loss.
I also lash a West Marine inflatable PFD around the deck of my boat. If I ever flip it and if I find that it is too water logged to turn over (my present ghetto yak does not have separate/dry storage compartments so it takes on lots of water) I figure that the PFD will help me right the boat and/or at lest get me to a point where I can use my pump to get the water out of the boat. Obviously I also wear a PFD.
Has anyone had experience swimming in waders while wearing a PFD?