I'm not a parent, but I have thought much about this... and I think the hardest thing would be for a child to not be walking with the Lord. 3 John 1:4 says,
I have no greater joy than this, to hear of my children walking in the truth. I think for this not to be the case would be the greatest sorrow for a parent. As tough as the death of a child would be, there is great peace in knowing that they are with the Lord, but for one that isn't walking with Him, there isn't much peace. It would be very tough.

Isaiah 40:28-31
Resolved, never to do anything, which I should be afraid to do, if it were the last hour of my life.
-Jonathan Edwards-