This post describes a "HTTP server" for the Arduino (or similar processors). It is specifically designed to work in very tight memory situations. Features are:
- Minimal memory (RAM) requirements (about 256 bytes)
- Small code size (around 2.5 KB)
- No use of the String class, or dynamic memory allocation (to avoid heap fragmentation)
- Incoming HTTP (client) requests decoded "on the fly" by a state machine
- Handles all of:
- Request type (eg. POST, GET, etc.)
- Path (eg. /server/foo.htm)
- GET parameters (eg. /server/foo.htm?device=clock&mode=UTC)
- Header values (eg. Accept-Language: en, mi)
- Cookies (as sent by the web browser)
- POST data (ie. the contents of forms)
- Doesn't care what Ethernet library you are using - you call your Ethernet library, and send a byte at at time to the HTTP library.
- Compact and fast
Traditional web servers assume they have a reasonable amount of memory to hand (as they normally would, with modern PCs) and "batch up" headers, cookies, etc. in memory, and then call a client script (eg. a PHP script) once everything has been received. However this can mean large amounts of memory need to be devoted to strings (eg. cookies).
The "state machine" approach, on the other hand, detects things the moment they arrive, and sorts them into key/value pairs. For example:
/server/foo.htm?device=clock&mode=UTC
That URL consists of two GET parameters, namely:
device = clock
mode = UTC
By decoding on the fly, the library can call a user-supplied function (one you provide) so that the function would get the key (eg. "device") and the value (eg. "clock") and act on it immediately, if wanted.
Then that string data can be discarded, keeping memory usage very low.
Structure of HTTP client messages
To understand what the library is doing, you need to understand what the HyperText Transfer Protocol (HTTP) is doing.
Web clients (ie. web browsers such as Firefox, Chrome, Internet Explorer, etc) connect to a web server (such as Apache), usually on port 80, using TCP, and send an HTTP header, followed optionally by other text. It looks something like this:
POST /forum/showpost.php?id=12345&page=2 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:39.0)
Host: www.example.com
Cookie: foo=bar;
Accept-Language: en, mi
action=save&user_id=1234
Breaking down this into its component parts:
The HTTP server library processes each of those parts as it reaches them and then calls a function you supply to handle that part.
To do this you derive a custom class from the library class, overriding the virtual functions in it, so you can write your own handlers, like this:
// derive an instance of the HTTPserver class with custom handlers
class myServerClass : public HTTPserver
{
virtual void processPostType (const char * key, const byte flags);
virtual void processPathname (const char * key, const byte flags);
virtual void processHttpVersion (const char * key, const byte flags);
virtual void processGetArgument (const char * key, const char * value, const byte flags);
virtual void processHeaderArgument (const char * key, const char * value, const byte flags);
virtual void processCookie (const char * key, const char * value, const byte flags);
virtual void processPostArgument (const char * key, const char * value, const byte flags);
}; // end of myServClass
myServerClass myServer; // instance of this class
You don't have to provide all those functions. Any you omit will silently ignore that part of the HTTP message. For example, you may not care what the HTTP version is.
As an example, the processPostType function will get the word POST, GET, or similar.
Since this signals the start of the transaction with the client, you could respond with a simple header, if you were planning to send debugging information. For example:
void myServerClass::processPostType (const char * key, const byte flags)
{
println(F("HTTP/1.1 200 OK"));
println(F("Content-Type: text/plain"));
println(); // end of headers
print (F("GET/POST type: "));
println (key);
} // end of processPostType
Flags
The "flags" parameter is a bit mask which conveys extra information (you can ignore it). Possible bit masks are:
- FLAG_NONE --> no problems
- FLAG_KEY_BUFFER_OVERFLOW --> the key was truncated
- FLAG_VALUE_BUFFER_OVERFLOW --> the value was truncated
- FLAG_ENCODING_ERROR --> %xx encoding error
For example, if the flag FLAG_VALUE_BUFFER_OVERFLOW is set, then some of the value was truncated, because it would not fit into the holding buffer.
eg.
if (flags & FLAG_VALUE_BUFFER_OVERFLOW)
{
// the value was too large to fit
}
POST type
This would normally be the word GET or POST (in uppercase). It is the start of the transaction and could be used to set up a response. Note that if you are planning to set cookies in response to POST data (which arrives later) you should not send the blank line which indicates the end of the headers.
Pathname
This is the "path" part of the URL, for example the text in bold and underlined:
http://somesite.com/forum/showpost.php?id=12938
In a normal web server this is the resource that we are "getting" - that is, the file that will be displayed, or the script that will be run.
In the example below we just echo back the pathname to the client:
void myServerClass::processPathname (const char * key, const byte flags)
{
print (F("Pathname: "));
println (key);
} // end of processPathname
GET arguments
Arguments on the URL line (GET arguments) are provided to your callback function, one at a time, in the order in which they appear.
For example the text in bold and underlined:
http://somesite.com/forum/showpost.php?id=12938&action=delete
In that example we would get two GET arguments:
- Name: id, Value: 12938
- Name: action, Value: delete
Example code that just echoes the GET arguments back:
void myServerClass::processGetArgument (const char * key, const char * value, const byte flags)
{
print (F("Get argument: "));
print (key);
print (F(" = "));
println (value);
} // end of processGetArgument
A more useful application would be to detect things like "light=on" or "fridge=off".
HTTP version
The final word on the first response line is the HTTP version. You can provide a handler if you want to discover what that is (eg. "HTTP/1.1").
void myServerClass::processHttpVersion (const char * key, const byte flags)
{
print (F("HTTP version: "));
println (key);
} // end of processHttpVersion
Header lines
Following the first line will be a series of header lines, for example identifying the web browser name.
An example would be:
Host: 10.0.0.241
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:39.0) Gecko/20100101 Firefox/39.0
Accept: text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Cookie: action=timetodance;
Connection: keep-alive
For each header line (except the Cookie line, discussed below) your processHeaderArgument function will be called with a key/value pair. For example:
- Key: Connection, Value: keep-alive
- Key: Accept-Encoding, Value: gzip, deflate
Example code that just echoes the header lines back:
void myServerClass::processHeaderArgument (const char * key, const char * value, const byte flags)
{
print (F("Header argument: "));
print (key);
print (F(" = "));
println (value);
} // end of processHeaderArgument
Cookies
Cookies are pieces of text stored at the client end (the web browser) and automatically sent to the server when you connect to it. The cookies are sent per URL, so for example my testing with 10.0.0.241 gives me only the cookies in the web browser identified by that URL.
The library identifies the "Cookie:" header, and breaks up the cookie header further into individual cookies (you do not get the entire Cookie header line).
For each cookie your processCookie function is called.
void myServerClass::processCookie (const char * key, const char * value, const byte flags)
{
print (F("Cookie: "));
print (key);
print (F(" = "));
println (value);
} // end of processCookie
This gives you the opportunity to find information you had previously sent to the client (using setCookie) in order to main a "memory" from one page to the next.
Note that if you are planning to set cookies they have to be set as part of the header returned to the client, that is, before any HTML is sent.
POST arguments
Finally, if this is a POST request (that is, not a GET request) there may be POST arguments. Typically these are because of a form on the web page.
A simple form might look like this:
<form METHOD="post" ACTION="/activate_leds">
<br>LED: 3 <input type=checkbox name="led_3" value=1 >
<br>LED: 4 <input type=checkbox name="led_4" value=1 >
<br>LED: 5 <input type=checkbox name="led_5" value=1 >
<p><input Type=submit Name=Submit Value="Process">
</form>
On the web browser that would look like:
For each form input type (which is not zero) you will get a call to processPostArgument. So in this case if the user checked LED 3 and LED 5 you would get processPostArgument called for those two inputs. The name would be "led_3" (for example) and the value would be "1" in this case.
void myServerClass::processPostArgument (const char * key, const char * value, const byte flags)
{
if (memcmp (key, "led_", 4) == 0 && isdigit (key [4]) )
{
int which = atoi (&key [4]);
if (which >= LOW_PIN && which <= HIGH_PIN)
digitalWrite (which, HIGH);
}
} // end of processPostArgument
Speed and memory usage
One of my test programs (that turns on LEDs based on a form) returns this information:
Binary sketch size: 13,662 bytes (of a 32,256 byte maximum)
...
Free memory = 1413 bytes
Time taken = 198 milliseconds.
Thus we can see that there is still plenty of program memory (flash memory) over to do other things. There is also plenty of RAM over (1413 / 2048 bytes). Also the page was processed in about 1/5 of a second.
The state machine is fast because it is mainly working with single bytes (there are very few string comparisons) and at a particular moment, only one "state path" is taken, which minimizes CPU usage.
It is also compact because it only ever saves the current key/value pair, before calling the callback routine and then discarding it.
Putting it all together - minimal example
This minimal example calls the HTTP library, without overriding any functions, and thus is a minimal "do nothing" example. However it does return a response to the client.
// Tiny web server demo
// Author: Nick Gammon
// Date: 20 July 2015
#include <SPI.h>
#include <Ethernet.h>
#include <HTTPserver.h>
// Enter a MAC address and IP address for your controller below.
byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x2D, 0xA1 };
// The IP address will be dependent on your local network:
byte ip[] = { 10, 0, 0, 241 };
// the router's gateway address:
byte gateway[] = { 10, 0, 0, 1 };
// the subnet mask
byte subnet[] = { 255, 255, 255, 0 };
// Initialize the Ethernet server library
EthernetServer server(80);
// derive an instance of the HTTPserver class with custom handlers
class myServerClass : public HTTPserver
{
}; // end of myServerClass
myServerClass myServer;
void setup ()
{
// start the Ethernet connection and the server:
Ethernet.begin(mac, ip, gateway, subnet);
server.begin();
} // end of setup
void loop ()
{
// listen for incoming clients
EthernetClient client = server.available();
if (!client)
{
// do other processing here
return;
}
myServer.begin (&client);
while (client.connected() && !myServer.done)
{
while (client.available () > 0 && !myServer.done)
myServer.processIncomingByte (client.read ());
// do other stuff here
} // end of while client connected
client.println(F("HTTP/1.1 200 OK"));
client.println(F("Content-Type: text/plain"));
client.println();
client.println(F("OK, done."));
// give the web browser time to receive the data
delay(1);
// close the connection:
client.stop();
} // end of loop
The important part is these lines which "feed" incoming data into the state machine:
myServer.begin (&client);
while (client.connected() && !myServer.done)
{
while (client.available () > 0 && !myServer.done)
myServer.processIncomingByte (client.read ());
// do other stuff here
} // end of while client connected
As the comment indicates, while there is no outstanding data, you can do other things, like take measurements. Once the incoming data is terminated (or the client disconnected) then you leave this processing loop.
The "myServer.begin" line passes to the library the current ethernet client address. This lets the callback functions send responses to the client by doing print or println function calls.
Provide callback functions
The example below overrides all of the callback functions, and then echoes back all the information to the client. If you run this you should see information about your web browser.
// Tiny web server demo
// Author: Nick Gammon
// Date: 20 July 2015
#include <SPI.h>
#include <Ethernet.h>
#include <HTTPserver.h>
// Enter a MAC address and IP address for your controller below.
byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x2D, 0xA1 };
// The IP address will be dependent on your local network:
byte ip[] = { 10, 0, 0, 241 };
// the router's gateway address:
byte gateway[] = { 10, 0, 0, 1 };
// the subnet mask
byte subnet[] = { 255, 255, 255, 0 };
// Initialize the Ethernet server library
EthernetServer server(80);
// derive an instance of the HTTPserver class with custom handlers
class myServerClass : public HTTPserver
{
virtual void processPostType (const char * key, const byte flags);
virtual void processPathname (const char * key, const byte flags);
virtual void processHttpVersion (const char * key, const byte flags);
virtual void processGetArgument (const char * key, const char * value, const byte flags);
virtual void processHeaderArgument (const char * key, const char * value, const byte flags);
virtual void processCookie (const char * key, const char * value, const byte flags);
virtual void processPostArgument (const char * key, const char * value, const byte flags);
}; // end of myServerClass
myServerClass myServer;
// -----------------------------------------------
// User handlers
// -----------------------------------------------
void myServerClass::processPostType (const char * key, const byte flags)
{
println(F("HTTP/1.1 200 OK"));
println(F("Content-Type: text/plain"));
println();
print (F("GET/POST type: "));
println (key);
} // end of processPostType
void myServerClass::processPathname (const char * key, const byte flags)
{
print (F("Pathname: "));
println (key);
} // end of processPathname
void myServerClass::processHttpVersion (const char * key, const byte flags)
{
print (F("HTTP version: "));
println (key);
} // end of processHttpVersion
void myServerClass::processGetArgument (const char * key, const char * value, const byte flags)
{
print (F("Get argument: "));
print (key);
print (F(" = "));
println (value);
} // end of processGetArgument
void myServerClass::processHeaderArgument (const char * key, const char * value, const byte flags)
{
print (F("Header argument: "));
print (key);
print (F(" = "));
println (value);
} // end of processHeaderArgument
void myServerClass::processCookie (const char * key, const char * value, const byte flags)
{
print (F("Cookie: "));
print (key);
print (F(" = "));
println (value);
} // end of processCookie
void myServerClass::processPostArgument (const char * key, const char * value, const byte flags)
{
print (F("Post argument: "));
print (key);
print (F(" = "));
println (value);
} // end of processPostArgument
// -----------------------------------------------
// End of user handlers
// -----------------------------------------------
void setup ()
{
// start the Ethernet connection and the server:
Ethernet.begin(mac, ip, gateway, subnet);
server.begin();
} // end of setup
void loop ()
{
// listen for incoming clients
EthernetClient client = server.available();
if (!client)
return;
myServer.begin (&client);
while (client.connected() && !myServer.done)
{
while (client.available () > 0 && !myServer.done)
myServer.processIncomingByte (client.read ());
// do other stuff here
} // end of while client connected
client.println(F("OK, done."));
// give the web browser time to receive the data
delay(1);
// close the connection:
client.stop();
} // end of loop
You should call the begin function each time you get a new incoming connection, as that resets the state machine back to the start, ready for a new HTTP session.
Utility functions
fixHTML
This converts the special HTML characters < > and & into < > and &
This should be used if you are planning to echo back data which might contain those symbols. The function also sends the data to the client, to save having to allocate a temporary buffer to hold the converted string.
urlEncode
This "percent-encodes" a string so that things like spaces and other special characters get turned into hex, for example a space becomes %20. This also sends the converted string to the client.
setCookie
This emits a "Set-Cookie:" header, omitting characters (such as ";") that are not permitted in cookies.
Key/value sizes
There are two constants in the library controlling the maximum sizes of keys and values:
static const size_t MAX_KEY_LENGTH = 40; // maximum size for a key
static const size_t MAX_VALUE_LENGTH = 100; // maximum size for a value
You can make them larger if you expect larger keys (eg. header names) or larger values (eg. header values). However the larger you make them, the more RAM will be used.
General nature of HTTP response
The first things you send back to the web client are:
- A response line (eg. "HTTP/1.1 200 OK")
- One or more header lines, for example describing the nature of the response (text or HTML, etc.)
- A blank line, which separates the headers from the rest of the response
- The text or HTML data
For example, to send plain text:
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Server: HTTPserver/1.0.0 (Arduino)
Some text here
To send HTML:
HTTP/1.1 200 OK
Content-Type: text/html
Connection: close
Server: HTTPserver/1.0.0 (Arduino)
<!DOCTYPE html>
<html>
<head>
<title>Arduino test</title>
</head>
<body>
<h1> My heading </h1>
<p>Some text
</body>
</html>
Use the F() macro
My examples show extensive use of the F() macro, for example:
client.println(F("OK, done."));
You should use the F() macro for sending string constants, otherwise the string is copied into RAM. You will use up RAM very quickly if you have length amounts of HTML in your code, without using F().
Source code
Download the library from: https://github.com/nickgammon/HTTPserver
Install it into your "libraries" folder underneath your sketchbook folder. |
![[Go to top]](/images/gototop.gif){kind=small}
top