Been there myself. I think the first few years my kids were really little was the least fishing I did. Probably because my ex had NO mother skills, and don't know if the kids would've been here if I didn't stay home more (why I have full custody of the kids now).
Your days will come. Then they will be your fishing buddies. Mine are dying for me to take them out. Hopefully soon. But will wait when good #'s are in and have access to a boat (since all my good motor boats are gone). But, in the end, they'll make your fishing better.
"Good angling ethic is not a monopoly practiced just by fly fisherman" - Bill McMillan