Different appoach for HTTP Client directly using the Stream, alternative to HTTPClient

[Jeroen88]

ClientHTTP

The class ClientHTTP implements support for REST API client calls to a HTTP server. It inherits from Client and thus implements a superset of that class' interface.

The class ClientHTTP decorates the underlying stream with HTTP request headers and reads the HTTP response headers from the stream before the stream is made available to the program. Also chunked responses are handled 'under the hood' offering a clean stream to which a request payload (if any) can be written using the standard Print functions like Print::write(), Print::print() and Print::printf(), and from which response payloads (if any) can be read using the standard Stream::read() functions.

The class ClientHTTP has a smaller memory footprint than the alternative class HTTPClient. Moreover: because of the Stream oriented nature of class ClientHTTP no memory buffers are needed, neither for sending payload, nor for receiving payload. It still remains possible to use such memory buffers though.
Chunked response from the stream is correctly implemented.
No Strings are used to prevent heap fragmentation. Only small stack buffers are used. Two exceptions are: the host name is copied into a heap char array, and the requestHeaders and responseHeaders are dynamic standard containers (map) using std::string for both key and value.

class ClientHTTP is capable of handling redirections. If not needed support for this may be omitted by removing the #define SUPPORT_REDIRECTS in the header file, thus saving some program memory bytes.

Request headers can easily be set by class ClientHTTP. Response headers can be easily collected by class ClientHTTP.

Any client may be used, e.g. WiFiClient, WiFiClientSecure and EthernetClient.

A HTTP GET, HEAD, POST, PUT, PATCH or DELETE always follows the same skeleton.

Integration with ArduinoJson, version 5 and 6, both for sending and receiving Json payloads is very intuitive.

The same library is defined for both the ESP8266 and the ESP32.

The following examples are available. All but one are HTTPS / TLS 1.2 examples:

• Authorization: how to use basic authorization
• BasicClientHTTP: how to GET a payload from a HTTP server and print it to Serial
• BasicClientHTTPS: how to GET a payload from a HTTPS server and print it to Serial
• DataClientHTTPS: how to GET a payload from a HTTPS server and handle the data using StreamString or a uint8_t buffer[]
• POSTClientHTTPS: how to POST a payload to a HTTPS server and print the response to Serial
• POSTJsonClientHTTPS: how to POST an ArduinoJson to a HTTPS server and deserialize the response into another ArduinJson without buffers. All data is buffered into the ArduinoJson only
• POSTSizerClientHTTPS: how to determine the size of a payload by printing to a utility class Sizer first. Next the resulting length is used to set the Content-Length request header by calling POST
• ReuseConnectionHTTPS: how to reuse a connection, including correct handling of the Connection: close response header. This example DOES NOT WORK on the ESP8266 YET!

Still to be done:

• Support for redirection to a different host
• Fix the ReuseConnectopnHTTPS example for the ESP8266

Request and response headers

Request headers can be set by just defining them before calling the REST method (GET, POST, etc):

  http.requestHeaders["Content-Type"] = "application/json";
  http.requestHeaders["Connection"] = "close";

Response headers can be collected by just defining them before ::status() is called:

  http.responseHeaders["Content-Type"];

After the ::status() command responseHeaders['Content-Type'] is either an empty string if the server did not send the response header, or it is populated with the value sent.

Skeleton program

A HTTP REST API call always follows the following skeleton

• Prepare a transport client, like WiFiClient client or WiFiClientSecure client
• Prepare a ClientHTTP http(client), passing the transport client into the constructor
• Set request headers
• Define response headers to be collected
• htpp.connect(host, port)
• Call the REST API method, e.g. http.POST(payload, payloadLength)
• Send the payload, if any, to the server using standard commands like Print::write(), Print::print() and Print:printf()
• Call http.status() which reads the response headers from the stream and returns the HTTP response code (greater than 0) or an error code (less than 0)
• Check any response headers, if needed
• Read the payload sent by the server, if any, using standard Stream::read() commands
• Close the connection calling http.stop(), or reuse the connection if the server did not sent a Connection: close response header, nor closed it's side of the connection

ArduinoJson integration

ArduinoJson documents, both version 5 and version 6, can directly be POSTed to a REST server:

  DynamicJsonDocument requestDocument(requestCapacity);
  ...
  http.requestHeaders["Content-Type"] = "application/json";
  http.POST("/post", measureJson(requestDocument));
  serializeJson(requestDocument, http);
  ...

Also ArduinoJson documents, both version 5 and 6, can directly be deserialized from the response:

  ...
  http.responseHeaders["Content-Type"];
  http.status();
  DynamicJsonDocument responseDocument(responseCapacity);
  if(strcasecmp(http.responseHeaders["Content-Type"], "application/json") == 0) {
    deserializeJson(responseDocument, http);
  }

See the examples for how to use ArduinoJson 5 .

[Jeroen88]

Checks failed for PlaformIO on ubuntu-latest with ##[error]Process completed with exit code 1, how can I fix this?

[Stanimir-Petev]

@Jeroen88 , have the same issue with my pull request. And I have simply added upload speed options for 4 boards.

[Jeroen88]

@Jeroen88 , have the same issue with my pull request. And I have simply added upload speed options for 4 boards.

How did you do that?

[Stanimir-Petev]

Do you mean the upload speed options? If so - I just added in the file boards.txt submenu for the respective boards with the options (basically I copied it from the other boards). you can check it out:
https://github.com/espressif/arduino-esp32/pull/3849/files#diff-fce09ad94aad2ba67472cd9142f00146

[Jeroen88]

Do you mean the upload speed options? If so - I just added in the file boards.txt submenu for the respective boards with the options (basically I copied it from the other boards). you can check it out:
https://github.com/espressif/arduino-esp32/pull/3849/files#diff-fce09ad94aad2ba67472cd9142f00146

Thnx for the advise, but this did not solve the check problem. Your PR #3849 has the same problem

[Stanimir-Petev]

Yes, that's what I said, but now I realized that it sounded like it solved the problem (which is not the case). :) I still look for a solution :)

[Jeroen88]

Yes, that's what I said, but now I realized that it sounded like it solved the problem (which is not the case). :) I still look for a solution :)

If you have a solution please share! :)

UPDATE: solution in next commit (in file install-platformio-esp32.sh)

[lbernstone]

Does this conform to the Arduino API such that it would be a replacement for HTTPClient if the names were changed? If not, it should be separated into its own repo and submitted to arduino for inclusion as a 3rd party library.

[Jeroen88]

@lbernstone thnx for your suggestion. I never realized that the Arduino API should be followed to have a library added to this repo. No, it has a different API. I might follow your suggestion and submit it to Arduino. Closing for now.

[lbernstone]

I don't speak for espressif or @me-no-dev, but my feeling is that since Arduino has a good method for 3rd party extensions, this codebase should adhere pretty strictly to providing the core Arduino API. The existing HTTPClient has some major issues, so it would be nice to get a compatible replacement 😄

[Jeroen88]

@lbernstone I think it should be relatively simple to make a wrapper around it that delivers the Arduino API, but it is quite some time ago that I created this library (although I am using is every day without any issues for WiFiClientSecure and a mobile NB-IoT client). Could you give the examples a try and tell me what you think? Especially if it is worth the effort to write a wrapper in your opinion?

[lbernstone]

I think it is far more important to write a good library than to force it into an API written 10 years ago. I'm not the best person to test this, but will direct people having issues to your code.