Add unresolved local service address recovery

[ddelong]

Problem

Sometimes after a deploy, resource managers get stuck in a state where they can't support specific job types that require HDFS delegation. It manifests into this error in those jobs:

Cannot invoke "java.net.InetAddress.isAnyLocalAddress()" because the return value of "java.net.InetSocketAddress.getAddress()" is null

Recycling the resource managers seems to resolve the condition.

See also: https://git.hubteam.com/HubSpot/HadoopPlanning/issues/326

Solution

My intuition of the problem here is that domain name in these objects isn't resolving at the time of creation. We have no explanation for that condition currently, but this code gives us a way to confirm or deny that while also improving the situation for customers. We reconstruct the InetSocketAddress to force resolution of the domain name so the getAddress() call doesn't return null. Also provide logging to gather data on how often this happens and if this recovers as expected.

Testing

We will want to test this in a few clusters to verify no regressions, but otherwise this should be benign.

[johnnysohn]

🚢