I would visit Orange Creek for the fishing but also because it's a great drive and you'll see alot of the island. You could also check out Alligator Point on the same day. For that matter doing a quick drive by of Pidgeon Bay Beach Club would be worth it if only to consider it for a future trip. Don't miss Mount Alvernia Hermitage. The climb up is great and the stone building is incredible.
Also check out Fernadez Bay for dinner. It's a really nice place.
I PM'd you with additional info.