Fixed-width record sent as delimited data

Fixed-width record sent as delimited data lives in the glue between systems: pools, balancers, DNS caches, and retry logic. Each one fails in its own boring way.

What it means

• A connection pool reuses sockets to hide connect cost; when it is exhausted, new requests queue or fail even though the backend is healthy.
• Load balancers only route around failures they can detect - a shallow health check keeps sending traffic to a dead endpoint.
• DNS TTL caching means clients keep using an old address long after you moved the service.

Common causes

• Pool sized for average load and exhausted at peak, or leaked connections never returned to the pool.
• Health check hitting a shallow URL that returns 200 while the real backend behind it is down.
• DNS change made while clients cache the old record for the full TTL.
• Retry policy with no backoff hammering a struggling endpoint and turning a blip into an outage.
• Credential or certificate rotated on one side but not the other.
• MQ bridge or messaging link adding latency the synchronous caller was never designed for.

How to diagnose

• Check pool metrics at failure time: active versus idle versus waiting threads.
• Query the load balancer pool state to see which members it considers healthy.
• Compare nslookup against the client resolver with a query to the authoritative server to spot stale cache.
• Correlate retry counts in the client log with backend request counts - a multiplier means retries are amplifying the load.
• Verify certificate expiry dates on both ends before assuming a network fault.

How to fix it

• Size pools for peak plus headroom, and set timeouts so leaked connections get reclaimed.
• Make health checks exercise the real backend path, not a static page.
• Lower DNS TTL before a migration and restore it after the cutover.
• Add exponential backoff with jitter and a retry budget; make mutating calls idempotent.

Example: Spot a stale DNS cache

Compare the client resolver answer with the authoritative server answer; a mismatch is a caching problem.

Interview tip

• Q: After a DNS change, half your clients still hit the old mainframe IP. Fixed-width record sent as delimited data - why, and what do you do?
• A: The clients are honoring the old TTL and caching the previous record. I confirm it with a query against the authoritative server, then either wait out the TTL or flush the client caches where I control them. For the next migration I lower the TTL well in advance so the cutover propagates quickly.

Prevention checklist

• Monitor pool utilization and alert before exhaustion, not after.
• Health checks must test the real transaction path end to end.
• Keep TTLs low for endpoints that move, and document every credential rotation date.
• Give every mutating API an idempotency key so retries are safe by design.