Skip to content

Different appoach for HTTP Client directly using the Stream, alternative to HTTPClient #3848

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 17 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Change the map key/value pair to <std::string, std::string>
  • Loading branch information
Jeroen88 committed Mar 25, 2020
commit 65d3032b23719deabc2d7ed91e58a17b45d74f1b
76 changes: 24 additions & 52 deletions libraries/ClientHTTP/README.md
Original file line number Diff line number Diff line change
@@ -1,62 +1,39 @@
# ClientHTTP
The ClientHTTP class implements support for REST API client calls to a HTTP server.
It inherits from Client and thus implements a superset of that class' interface.

The ClientHTTP class decorates the underlying stream with HTTP request headers and
reads the HTTP response headers from the stream before the stream is made available to
the program. Also chunked responses are handled 'under the hood' offering a clean stream
to which a request payload (if any) can be written using the standard Print::write()
functions and from which response payloads (if any) can be read using the standard
Stream::read() functions.

The ClientHTTP class has a smaller memory footprint than the alternative HTTPClient class.
Moreover: because of the Stream oriented nature of ClientHTTP no memory buffers are
needed, neither for sending payload, nor for receiving payload. It still remains possible
to use such memory buffers though. Chunked response from the stream is correctly implemented.
No Strings are used to prevent heap fragmentation. Only small stack buffers are used. Two
exceptions are: the host name is copied into a heap char array, and the requestHeaders and
responseHeaders are dynamic standard containers (map).

ClientHTTP is able to handle redirections, but support for this may be omitted by clearing
the #define SUPPORT_REDIRECTS in the header file, thus saving some programm memory bytes.
The ClientHTTP class implements support for REST API client calls to a HTTP server. It inherits from Client and thus implements a superset of that class' interface.

The ClientHTTP class decorates the underlying stream with HTTP request headers and reads the HTTP response headers from the stream before the stream is made available to the program. Also chunked responses are handled 'under the hood' offering a clean stream to which a request payload (if any) can be written using the standard Print::write() functions and from which response payloads (if any) can be read using the standard Stream::read() functions.

The ClientHTTP class has a smaller memory footprint than the alternative HTTPClient class. Moreover: because of the Stream oriented nature of ClientHTTP no memory buffers are needed, neither for sending payload, nor for receiving payload. It still remains possible to use such memory buffers though. Chunked response from the stream is correctly implemented.
No Strings are used to prevent heap fragmentation. Only small stack buffers are used. Two exceptions are: the host name is copied into a heap char array, and the requestHeaders and responseHeaders are dynamic standard containers (map) with `std::string` key and `std::string` value.

ClientHTTP is able to handle redirections. Support for this may be omitted by clearing the `#define SUPPORT_REDIRECTS` in the header file, thus saving some program memory bytes.

Request headers can easily be set by ClientHTTP. Response headers can be easily collected by
ClientHTTP.

Any client may be used, e.g. WiFiClient, WiFiClientSecure and EthernetClient.
Any client may be used, e.g. `WiFiClient`, `WiFiClientSecure` and `EthernetClient`.

A HTTP GET, HEAD, POST, PUT, PATCH or DELETE always follows the same skeleton.
A HTTP `GET`, `HEAD`, `POST`, `PUT`, `PATCH` or `DELETE` always follows the same skeleton.

Integration with ArduinoJson, version 5 and 6, both for sending and receiving Json payloads
is very intuitive.
Integration with ArduinoJson, version 5 and 6, both for sending and receiving Json payloads is very intuitive.

The same library is defined for both the ESP8266 and the ESP32.

The following examples are available. All but one are HTTPS / TLS 1.2 examples:
- Authorization: how to use basic authorization
- BasicClientHTTP: how to GET a payload from a HTTP server and print it to Serial
- BasicClientHTTPS: how to GET a paylaod from a HTTPS server and print it to Serial
- DataClientHTTPS: how to GET a payload from a HTTPS server and handle the data using
StreamString or a buffer
- POSTClientHTTPS: how to POST a payload to a HTTPS server and print the response to
Serial
- POSTJsonClientHTTPS: how to POST an ArduinoJson to a HTTPS server and deserialize the
response into another ArduinJson without buffers. All data is buffered into the
ArduinoJson only
- POSTSizerClientHTTPS: how to determine the size of a payload by printing to a utility
class Sizer first. Next the resulting length is used to set the Content-Length request
header by calling POST
- ReuseConnectionHTTPS: how to reuse a connection, including correct handling of the
Connection: close response header. This example DOES NOT WORK on the ESP8266 YET!
- BasicClientHTTP: how to `GET` a payload from a HTTP server and print it to `Serial`
- BasicClientHTTPS: how to `GET` a paylaod from a HTTPS server and print it to `Serial`
- DataClientHTTPS: how to `GET` a payload from a HTTPS server and handle the data using `StreamString` or a buffer
- POSTClientHTTPS: how to `POST` a payload to a HTTPS server and print the response to `Serial`
- POSTJsonClientHTTPS: how to `POST` an ArduinoJson to a HTTPS server and deserialize the response into another ArduinJson without buffers. All data is buffered into the ArduinoJson only
- POSTSizerClientHTTPS: how to determine the size of a payload by printing to a utility `class Sizer` first. Next the resulting length is used to set the Content-Length request header by calling `POST`
- ReuseConnectionHTTPS: how to reuse a connection, including correct handling of the Connection: close response header. This example DOES NOT WORK on the ESP8266 YET!

Still to be done:
- Support for redirection to a different host
- Fixing the response header handling. I think an Allocator needs to be defined for this
but I could use some help on this one :)

## Request and response headers
Request headers can be set by just defining them before calling the REST method (`GET`,
`POST`, etc):
Request headers can be set by just defining them before calling the REST method (`GET`, `POST`, etc):
```cpp
http.requestHeaders["Content-Type"] = "application/json";
http.requestHeaders["Connection"] = "close";
Expand All @@ -65,8 +42,7 @@ Response headers can be collected by just defining them before `::status()` is c
```cpp
http.responseHeaders["Content-Type"];
```
After the `::status()` command `responseHeaders['Content-Type']` is either `NULL` if the server
did not send the response header, or it is populated with the value sent.
After the `::status()` command `responseHeaders['Content-Type']` is either an empty string if the server did not send the response header, or it is populated with the value sent.

## Skeleton program
A HTTP REST API call always follows the following skeleton
Expand All @@ -77,13 +53,10 @@ A HTTP REST API call always follows the following skeleton
- `htpp.connect(host, port)`
- Call the REST API method, e.g. `http.POST(payload, payloadLength)`
- Send the payload, if any, to the server using standard `Print::write()` commands
- Call `http.status()` which reads the response headers from the stream and returns
the HTTP response code (greater than 0) or an error code (less than 0)
- Call `http.status()` which reads the response headers from the stream and returns the HTTP response code (greater than 0) or an error code (less than 0)
- Check any response headers, if needed
- Read the payload sent by the server, if any, using standard `Stream::read()` commands
- Close the connection calling `http.stop()`, or reuse the connection if the server
did not sent a `Connection: close` response header, nor closed it's side of the
connection
- Close the connection calling `http.stop()`, or reuse the connection if the server did not sent a `Connection: close` response header, nor closed it's side of the connection

## ArduinoJson integration
ArduinoJson documents, both version 5 and version 6, can directly be POSTed to a REST server:
Expand All @@ -96,8 +69,7 @@ ArduinoJson documents, both version 5 and version 6, can directly be POSTed to a
serializeJson(requestDocument, http);
...
```
Also ArduinoJson documents, both version 5 and 6, can directly be deserialized from the
response:
Also ArduinoJson documents, both version 5 and 6, can directly be deserialized from the response:
```cpp
...
http.responseHeaders["Content-Type"];
Expand All @@ -107,6 +79,6 @@ response:
deserializeJson(responseDocument, http);
}
```

See the examples for how to use ArduinoJson 5.


Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ void setup() {

String credentials = base64::encode((const uint8_t *)"guest:guest", sizeof "guest:guest" - 1);
String auth = String("Basic ") + credentials;
http.requestHeaders["Authorization"] = (char *)auth.c_str();
http.requestHeaders["Authorization"] = auth.c_str();
http.requestHeaders["Connection"] = "close";

if (!http.connect("jigsaw.w3.org", 443)) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,8 @@ void setup() {
Serial.println("Payload succesfully send!");
}

Serial.printf("Response header \"Content-Type: %s\"\n", http.responseHeaders["Content-Type"].c_str());

const size_t responseCapacity = 2*JSON_ARRAY_SIZE(2) + 3*JSON_OBJECT_SIZE(0) + 2*JSON_OBJECT_SIZE(3) + JSON_OBJECT_SIZE(6) + JSON_OBJECT_SIZE(7) + 618;
#if ARDUINOJSON_VERSION_MAJOR == 6
DynamicJsonDocument responseDocument(responseCapacity);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,8 @@ void loop() {
Serial.printf("Request failed with error code %d\n", status);
}

if(http.responseHeaders["Connection"] != NULL && strcasecmp(http.responseHeaders["Connection"], "close") == 0) {
Serial.printf("Response header \"Connection: %s\"\n", http.responseHeaders["Connection"].c_str());
if(strcasecmp(http.responseHeaders["Connection"].c_str(), "close") == 0) {
Serial.println("Closing connection because the server requested to do so");
http.stop();
} else {
Expand Down
11 changes: 7 additions & 4 deletions libraries/ClientHTTP/src/ClientHTTP.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -400,9 +400,11 @@ bool ClientHTTP::writeRequestHeaders(const char * method, const c
}

for(auto requestHeader: requestHeaders) { // std::pair<char *, char *>
_client.printf("%s: %s\r\n", requestHeader.first, requestHeader.second);
// _client.printf("%s: %s\r\n", requestHeader.first, requestHeader.second);
_client.printf("%s: %s\r\n", requestHeader.first.c_str(), requestHeader.second.c_str());
#if defined(ESP32)
log_v("Request header '%s: %s'", requestHeader.first, requestHeader.second);
// log_v("Request header '%s: %s'", requestHeader.first, requestHeader.second);
log_v("Request header '%s: %s'", requestHeader.first.c_str(), requestHeader.second.c_str());
#endif
}

Expand Down Expand Up @@ -605,8 +607,8 @@ ClientHTTP::http_code_t ClientHTTP::status() {


ClientHTTP::http_code_t ClientHTTP::readResponseHeaders() {

char headerLine[RESPONSE_HEADER_LINE_SIZE];

size_t bytesRead = _client.readBytesUntil('\n', headerLine, sizeof headerLine - 1);
headerLine[bytesRead] = '\0';
char * statusPtr = strchr(headerLine, ' ');
Expand Down Expand Up @@ -651,11 +653,12 @@ ClientHTTP::http_code_t ClientHTTP::readResponseHeaders() {
log_v("Read response header '%s: %s'", headerLine, valuePtr);
#endif

// Set response header TO DO
// Set response header
auto search = responseHeaders.find(headerLine);
if(search != responseHeaders.end()) {
#if defined(ESP32)
log_v("Set response header '%s: %s'", search->first, valuePtr);
search->second = valuePtr;
#endif
// free(responseHeader.second); // !!!!!! NOT SURE IF THESE POINTERS MUST BE FREE-ED
// responseHeader.second = strdup(valuePtr);
Expand Down
14 changes: 9 additions & 5 deletions libraries/ClientHTTP/src/ClientHTTP.h
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
*/

// TO DO
// map char * allocation and free. responseHeaders are not collected yet
// redirect to a different host
// Possibly add gzip stream

Expand Down Expand Up @@ -218,15 +217,20 @@ class ClientHTTP: public Client {

void printTo(Print &printer);

std::map<char *, char *> requestHeaders;
// std::map<char *, char *> requestHeaders;
std::map<std::string, std::string> requestHeaders;

// Helper struct for case insensitive comparison of keys in the response headers map
struct cmp_str {
bool operator()(char const *a, char const *b) const {
return strcasecmp(a, b) < 0;
// bool operator()(char const *a, char const *b) const {
// return strcasecmp(a, b) < 0;
// }
bool operator()(const std::string & a, const std::string & b) const {
return strcasecmp(a.c_str(), b.c_str()) < 0;
}
};
std::map<char *, char *, cmp_str> responseHeaders;
// std::map<char *, char *, cmp_str> responseHeaders;
std::map<std::string, std::string, cmp_str> responseHeaders;

typedef enum {
HTTP_10,
Expand Down