ClientConnectionFailure at forward-request

Issue

I have an Angular Web Application, that is backed by a C# Web Api, which facilitates speaking to an Azure Function App.

An rough example flow is like the following:

• Angular Web App (press download with selected parameters) -> send GET request to API Management Service
• API Management Service makes call to a C# Web Api
• C# Web Api then responds back to the APIM, which in turn calls an Azure Function App to further process data from an external source
• Once a csv is ready, the data payload is downloaded in the browser where the Web App is open

For larger payloads, the download request fails with the following error in Application Insights:

"ClientConnectionFailure at forward-request"

This error occurs at exactly 2 minutes, every time, unless the payload is sufficiently small.

This lead me to believe that the Function App, which I understand as the client in this situation, is timing out, and cancelling the request.

But testing a GET with the exact same parameters through a local instance of the Azure Function App using Postman, the payload is successfully retrieved.

So the issue isn't the Azure Function App, because it did not time out in Postman as when using the WebApp.

This leads me to three different possibilities:

1. The C# WebApi is timing out and cancelling the request before the APIM can respond in full
2. The WebApp itself is timing out.
3. The internet browser (Chrome), is timing out. (Chrome has a hard unchangeable timeout of 5 minutes, so unlikely)

#1. To tackle the the first option, I upgraded the timeout of the HttpClient created in the relevant download action:

public aync Task<HttpResponseMessage> DownloadIt(blah)
{
    HttpClient client = getHttpClient();
    client.Timeout = TimeSpan.FromMilliseconds(Convert.ToDouble(600000)); // 10 minutes
    var request = new HttpRequestMessage(HttpMethod.Get, buildQueryString(blah, client.BaseAddress));
    return await client.SendAsync(request);
}

private HttpClient getHttpClient()
{
    return _httpClientFactory.CreateClient("blah");
}

This had no effect as the same error was observed.

#2. There are a couple of Timeout properties in the protractor.conf.js, like allScriptsTimeout and defaultTimeoutInterval.

Increasing these had no effect.

** There is a last possibility that the APIM itself is timing out, but looking into the APIM policy for the relevant API, there is no forward-request property, with a timeout, meaning by default according to Microsoft, there is no timeout for the APIM. https://docs.microsoft.com/en-us/azure/api-management/api-management-advanced-policies

I've tried a few different strategies but to no avail.

Solution

After testing each component of the solution locally (outside Azure), web app (front end), web api, function app (backend), it is clear that the issue was caused by Azure itself, namely the default 4 minutes for Idle Timeout at the Azure Load Balancer.

I double checked by timing the requests that failed and always got 4 minutes.

The way the code in the backend is sending requests is all together, for larger data sets this caused it to hit the load balancer's timeout.

It looks like the load balancer timeout is configurable, but this doesn't look like something I will be able to change.

So solution: Write more efficiet/better code in the backend.

Answered By - mkcoding