Think baitfish. Just because its offshore doesn't mean it has to be big.
For example, late in the year the sardinas down Cabo way are in the 1-2 inch range and anything bigger gets ignored. Small olive/white clousers with just a touch of flash fished slow and deep account for a lot of yellowfins to 25lb or so.
If they're chowing down on adult mackerel or ballyhoo then your offering will need to be bigger.
There are a few Harkers regulars here who can offer first hand experience.
When sight fishing, look over your shoulder from time to time, you never know who's behind you