Like most other sports, kayaking comes with lots of optional "extras", a rudder being one. I took a lesson first time out. On the advice of my instructor, I learned to paddle without a rudder and do fine in all conditions. I guess it's a bit like learning to drive on a stick shift car versus one with automatic transmission. That said, the shape of the hull has a lot to do with how the kayak tracks and behaves in wind.
Guys who kayak fish with conventional gear like them a lot because it makes tracking (basically staying on a straight course) a lot easier when trolling with multiple rigs.
Some of the more recent hulls designed specifically for fishing are pretty flat and would probably be hard work to keep on track without a rudder. Some folks swear by them, others (like me) make do without.
If I had one I would probably have wrecked it by now by forgetting to retract it when hitting the beach
Before you "take the plunge" (speaking both metaphorically and literally
) and buy a pungo (sit inside), I suggest looking at sit on top designs. I don't want to start a major sit-in versus sit-on debate (probably will) but here are my thoughts:
At some point you will capsize - it probably wont be your fault - there are a lot of brainless idiots on the water who won't see you 'till its too late. Getting back on a SOT is a piece of cake with a bit of practice.
Sit-in kayaks require a whole different set of skills. Early season when the water is cold you have to get back on the boat quickly.
Hope this helps a little bit.