Solved Empty response body from GET or POST HTTP request

Hi everyone,

I'm sending POST and GET http requests to a server via maxon::Url and maxon::FileUtilities::ReadFileToMemory(serverURL, charArray).
The charArray will be sucessfully filled with the response body if the server responds with a status code 200 (OK) after the request has been sent with ReadFileToMemory(serverURL, charArray) . If the server responds with an error code like 401 (Unauthorized) the ReadFileToMemory() function unfortunately also returns an error, even if the client / server - communication was successfull. The main problem is that the response body (charArray) will be empty in such a case, even if the response header ("content-length") shows that a response body with a specific size was sent. I need to process the content of the response body if the server responds with a status code other than 200 (OK) after a POST request, which is a common approach in web development.

Is the response body empty by design in such a case, or is this a bug?

I prepared a small example which sends a GET request to the service https://httpstat.us.
A request like https://httpstat.us/401 will respond with the body "401 Unauthorized" which will also be reflected in the response header:"Content-Length: 16".
This can be observed with a tool like Postman (https://www.postman.com/downloads/), but the example code below will unfortunately result in an empty response body.

Any ideas how I can get access to the response body? Thanks in advance!

Best regards,

Tim

#include "maxon/network_url.h"
#include "maxon/network_ip_ssl.h"
#include "maxon/network_ip.h"
#include "maxon/file_utilities.h"


static maxon::Result<void> ProducesAnEmptyResponseBody()
{
	maxon::Url serverURL{ String("https://httpstat.us/401") };

	// Set the request headers
	maxon::DataDictionary requestHeaders;
	requestHeaders.Set(maxon::Data(String("Content-Type")), maxon::Data(String("application/json"))) iferr_ignore();

	iferr (serverURL.Set(maxon::URLFLAGS::HTTP_ADDITIONAL_REQUEST_HEADER, requestHeaders))
	{
		ApplicationOutput("Error 1: @", err);
		return err;
	}

	// Setup a dictionary for the response headers
	maxon::DataDictionary responseHeaders;
	iferr (serverURL.Set(maxon::URLFLAGS::HTTP_RESPONSE_HEADER, &responseHeaders))
	{
		ApplicationOutput("Error 2: @", err);
		return err;
	}

	// Sends the request and retrieves the response in a memory buffer.
	// The response body will be stored in a char array.
	// The response headers (includes the http status code)
	// will be stored in the previously set response header dictionary.
	maxon::BaseArray<maxon::Char> charArray;
	iferr (maxon::FileUtilities::ReadFileToMemory(serverURL, charArray))
	{
		ApplicationOutput("Error 3: @", err);
	}

	// Store the response body as a string.
	String responseBody = maxon::String(charArray);
	ApplicationOutput("responseBody = '@'", responseBody);

        // Extract the status code from the response header, etc...
        // ApplicationOutput("responseHeaders = '@'", responseHeaders);

	return maxon::OK;
}

Hi @Braeburn,

well, it sort of depends on the http code. If you have a look at the RFC 2616 I did post above, there is not standardization whatsoever. You can technically return both a 404 and a list of your favorite movies in the content of a response. When I talked about error codes I meant the 4XX range. As you see in your Mozilla example page they at least slightly indicate that this is meant for a 200, i.e. when everything is okeydokey

I agree that the current Maxon solution is a bit weird and have also voiced that to the devs. For your problem on hand it mostly depends on the error code, the implementation of the server and the user agent for what you can expect in the body. So you would have to post an actual example, to get here any further. R21 however has left the support cycle of Maxon, so you won't get any fixes for it anymore.

But you could use Python as a glue language. The requests library I for example will return the "correct" body in your case here.

Cheers,
Ferdinand

import requests

for url in ["https://httpstat.us/401", 
            "https://httpstat.us/404", 
            "https://blog.fefe.de/test"]:
    print ("\n-", url, "-" * 20, "\n")
    response = requests.get(url)
    print (response)
    print (response.text)

- https://httpstat.us/401 -------------------- 

<Response [401]>
401 Unauthorized

- https://httpstat.us/404 -------------------- 

<Response [404]>
404 Not Found

- https://blog.fefe.de/test -------------------- 

<Response [404]>
<title>Not Found</title>
No such file or directory.

[Finished in 0.9s]

Hi @Braeburn,

thank you for reaching out to us. I'll unpack some stuff in bullet points:

1. Looking into the relevant RFC, you can see that there is very little standardisation, regarding what content/body you can expect for what message code.
2. The 4XX range all indicates an error with receiving the body of a request, so it is somewhat to be expected that you get no content here.
3. Some pages however have custom error pages and/or redirect some responses to 200 which explains the error.
4. When you inspect your url with the google chrome traffic monitor, you can see that there is no redirect happening, but that the 404 will indeed return some text content and the matching content length.

When you now shuffle through some http interfaces, google chrome (returns "correct" content and content length of 16), Python's urllib3 (returns no content and content length of 0), the request library (returns the same as chrome) and finally Cinema, you will see that they handle the responses quite differently. Which does not make much sense, since the client should have very little say in this. My suspicion would be that the user-agent of the request plays a role in this, or more specifically not setting up a proper agent, but I cannot confirm this yet. I will reach out to the development team to ask them, what their thoughts are about all that.

In the mean time you could just compose yourself the "content" for responses where the body is empty. The status code and message will in your case exactly equate to the "content" (the keys net.maxon.http.httpcode and net.maxon.http.httpmessage in the response header dictionary). This won't help you to catch the content of responses where there are custom error pages without redirects (something like this for example), but will get you pretty close.

Cheers,
Ferdinand

Hi @Braeburn,

I asked the devs and the solution is rather simple: The body of a response which indicates an error by its error code is wrapped into Cinema's error. Which leads in case of websites which use custom error pages without redirecting to them to some slightly awkward code. Below you will find an example for a 404 response which in fact returns some HTML in its body without using a redirect.

This approach is however only possible with R23 onwards, for R21 you have to compose your own stand in "body" out of the response header as I did indicate in my last posting and the code below. For your case the sum of the error code and the error message will equate exactly to the content length. For the case of html-error codes which also attach a full html body (without a redirect), like shown in my example below, you can only approximate the content of the body.

Cheers,
Ferdinand

#include "c4d.h"
#include "c4d_symbols.h"
#include "main.h"
#include "maxon/datatypelib.h"
#include "maxon/file_utilities.h"
#include "maxon/network_ip.h"
#include "maxon/network_ip_ssl.h"
#include "maxon/network_url.h"

// ...

maxon::Result<void> PC13041::ProducesAnEmptyResponseBody()
{
	iferr_scope;
        // The url which will return a 404 code and an HTML body.
	maxon::Url serverURL{ String("https://blog.fefe.de/test") };

	// Set the request headers
	maxon::DataDictionary requestHeaders;
	requestHeaders.Set(maxon::Data(String("Content-Type")), maxon::Data(String("application/json"))) iferr_return();

	iferr(serverURL.Set(maxon::URLFLAGS::HTTP_ADDITIONAL_REQUEST_HEADER, requestHeaders))
	{
		ApplicationOutput("Error 1: @", err);
		return err;
	}

	// Setup a dictionary for the response headers
	maxon::DataDictionary responseHeaders;
	iferr(serverURL.Set(maxon::URLFLAGS::HTTP_RESPONSE_HEADER, &responseHeaders))
	{
		ApplicationOutput("Error 2: @", err);
		return err;
	}

	// The HTML content/body will be wrapped into the error. 
	maxon::BaseArray<maxon::Char> charArray;
	iferr(maxon::FileUtilities::ReadFileToMemory(serverURL, charArray))
	{
		// Some stuff in the header of the response which might be useful to you.
		maxon::InternedId iidCode, iidMessage;
		iidCode.Init("net.maxon.http.httpcode"_s) iferr_return;
		iidMessage.Init("net.maxon.http.httpmessage"_s) iferr_return;
		const auto responseLength = responseHeaders.Get<maxon::String>("content-length"_s);
		const auto responseCode = responseHeaders.Get<Int64>(iidCode);
		const auto responseMessage = responseHeaders.Get<maxon::String>(iidMessage);

		ApplicationOutput("header-content-length: @", responseLength);
		ApplicationOutput("header-httpcode: @", responseCode);
		ApplicationOutput("header-httpmessage: @", responseMessage);

		// The body of a response that indicates by its error code that it has no body is fused with the
		// error message, so we have to do some string gymnastics to get it out there. This is certainly
		// not a very pretty solution, as it relies on hard-coded bits (i.e. is prone to errors). Using
		// at least a regex would be probably better here.
		const maxon::String message = err.GetMessage();
		maxon::BaseArray<maxon::String> parts;
		message.Split("Not FoundBody:"_s, true, parts) iferr_return;
		
		if (parts.GetCount() > 1) {
			ApplicationOutput("errorBody: @", parts[1]);
		}
		else {
			ApplicationOutput("errorMessage: @", message);
		}
	}

	String responseBody = maxon::String(charArray);
	ApplicationOutput("responseBody = '@'", responseBody);
	//ApplicationOutput("responseHeaders = '@'", responseHeaders);

	return maxon::OK;
}

Which will spit out:

header-content-length: 52
header-httpcode: 404
header-httpmessage: Not Found
errorBody: <title>Not Found</title>
No such file or directory.

Hi @zipit!

Thanks for the comprehensive reply!
Yes, your solution seems to produce a different result for R21. The body is unfortunately missing in the error message in R21:

header-content-length: 52
header-httpcode: 404
header-httpmessage: Not Found
errorMessage: invalid http response 404. Not Found (https://:[email protected]/test)

err.GetMessage(): invalid http response 404. Not Found (https://:[email protected]/test)

I need to get access to the (JSON encoded) response body because the server is responding with detailed informations about what exactly went wrong in different situations. Approximating this information from the httpcode and httpmessage is not sufficient in my case. I guess there is currently no solution for R21.

BTW: From my experience it is pretty common to process the response body in case of some errors. I processed such responses in JavaScript based clients and the responses came from .NET Core, Python FastAPI, Go and WordPress based servers. You might also get problems if you only count on a returned status code. As an example: In a JavaScript based client which uses fetch() (which is a living standard) to make a HTTP request, you would get a Response object with an attribute Response.status. Unfortunately Response.status is not supported on Chrome and Firefox on Android and you might also look at the response body in such a case. See: https://developer.mozilla.org/en-US/docs/Web/API/Response/status

Thank you very much!

Best regards,

Tim

Hi @Braeburn,

well, it sort of depends on the http code. If you have a look at the RFC 2616 I did post above, there is not standardization whatsoever. You can technically return both a 404 and a list of your favorite movies in the content of a response. When I talked about error codes I meant the 4XX range. As you see in your Mozilla example page they at least slightly indicate that this is meant for a 200, i.e. when everything is okeydokey

I agree that the current Maxon solution is a bit weird and have also voiced that to the devs. For your problem on hand it mostly depends on the error code, the implementation of the server and the user agent for what you can expect in the body. So you would have to post an actual example, to get here any further. R21 however has left the support cycle of Maxon, so you won't get any fixes for it anymore.

But you could use Python as a glue language. The requests library I for example will return the "correct" body in your case here.

Cheers,
Ferdinand

import requests

for url in ["https://httpstat.us/401", 
            "https://httpstat.us/404", 
            "https://blog.fefe.de/test"]:
    print ("\n-", url, "-" * 20, "\n")
    response = requests.get(url)
    print (response)
    print (response.text)

- https://httpstat.us/401 -------------------- 

<Response [401]>
401 Unauthorized

- https://httpstat.us/404 -------------------- 

<Response [404]>
404 Not Found

- https://blog.fefe.de/test -------------------- 

<Response [404]>
<title>Not Found</title>
No such file or directory.

[Finished in 0.9s]

Hi @zipit!

Our product supports multiple Cinema versions down to R16. We already have a working solution by using Python 2.x with urllib2. We wanted to switch to a pure Maxon based HTTP solution from R21 upwards. This worked out so far, except for the problem with the empty response body. The server responds with a 401 status code and it sends detailed informations about the failed authorization in the response body, which we would like to process. I'm aware this procedure is not based on an official standard but it is a very common procedure. So, we might need to use our Python based solution for R21 and S22 and switch to the Maxon solution for R23 and above.

Best regards,

Tim

Hi @Braeburn,

oh, I wasn't aware of that. Well, then there are really no new solutions in R21 for you.

Cheers,
Ferdinand

Hi,

without further feedback, we will consider this thread as solved by Monday and flag it accordingly.

Cheers,
Ferdinand