Handling web service requests using C, FastCGI and Lighttpd

Gagarine Yaikhom

18 December 2014, Thursday

This is further to the note on writing lean web services using C, FastCGI and MySQL. In this note, I discuss retrieving parameters from the HTTP request, and in the process discuss an alternative web server setup using lighttpd on GNU/Linux.

Install and configure lighttpd

First, install the lighttpd web server using the appropriate command on your system. For instance, on the Ubuntu distribution, you would run:

$ sudo apt-get install lighttpd

To configure a lighttpd server to use the FastCGI extension, refer to the lighttpd documentation. The configuration file on my system is /etc/lighttpd/lighttpd.conf. For the rest of this note, we shall use an example FastCGI application named echo, which we describe in the following sections. For now, we make this web service end-point accessible from the lighttpd server by adding the following to the configuration file:

fastcgi.debug = 1
fastcgi.server = (
    "/echo" => (
    "echo.fcgi.handler" => (
        "socket" => "/tmp/echo.fcgi.socket",
        "check-local" => "disable",
        "bin-path" => "/home/user/fcgiws/echo.fcgi",
        "max-procs" => 1
    )
)

In the above segment, we first enable FastCGI debugging. You can see the debugging output in the file pointed to by server.errorlog in the lighttpd configuration file above. For instance, on my system the error and logging messages are redirected to the following file.

server.errorlog = "/var/log/lighttpd/error.log"

In the second part, we configure an end-point that opens up a channel between the lighttpd server and our FastCGI application. We wish to make this application accessible at path echo relative to the server root. For instance, http://localhost/echo. We assign a label to this channel titled echo.fcgi.handler, in case we have multiple channels to various other FastCGI web service end-points.

The configuration for this channel is as follows:

Any request directed to /echo will be sent to the FastCGI process running the binary image stored at /home/user/fcgiws/echo.fcgi. We shall discuss creation, compilation and deployment of this example web service in the following sections.
For this channel, lighttpd will initialise only one process to handle all of the requests sent to this channel. This is specified using the "max-procs" => 1 setting.
By disabling the check-local flag, we are asking all requests to the channel to be directed immediately to the associated process without checking if the server.document-root exists.
Finally, we create a dedicated socket named /tmp/echo.fcgi.socket which lighttpd must use in order to pass request and response between the server and the FastCGI process.

Example application: compile and deploy

Our example application echo is a simple web service that returns a page with some of the variables received in the HTTP requests. In the note on writing lean web services using C, FastCGI and MySQL, I discussed accessing a MySQL database from a FastCGI application. In the following example, I wish to demonstrate how an application would extract the unique characteristics of each request, so that it can respond accordingly. For instance, retrieving the query parameters, or client user-agent information. With this facility at hand, our web services should be able to serve a wide range of clients and usage scenarios.

The following is the source code for echo:

#include <fcgi_stdio.h>
#include <stdlib.h>

/* some of the HTTP variables we are interest in */
#define MAX_VARS 30
char* vars[MAX_VARS] = {
    "DOCUMENT_ROOT",
    "GATEWAY_INTERFACE",
    "HTTP_ACCEPT",
    "HTTP_ACCEPT_ENCODING",
    "HTTP_ACCEPT_LANGUAGE",
    "HTTP_CACHE_CONTROL",
    "HTTP_CONNECTION",
    "HTTP_HOST",
    "HTTP_PRAGMA",
    "HTTP_RANGE",
    "HTTP_REFERER",
    "HTTP_TE",
    "HTTP_USER_AGENT",
    "HTTP_X_FORWARDED_FOR",
    "PATH",
    "QUERY_STRING",
    "REMOTE_ADDR",
    "REMOTE_HOST",
    "REMOTE_PORT",
    "REQUEST_METHOD",
    "REQUEST_URI",
    "SCRIPT_FILENAME",
    "SCRIPT_NAME",
    "SERVER_ADDR",
    "SERVER_ADMIN",
    "SERVER_NAME",
    "SERVER_PORT",
    "SERVER_PROTOCOL",
    "SERVER_SIGNATURE",
    "SERVER_SOFTWARE"
};

int main(void)
{
    int count = 0, i;
    char *v;
    while (FCGI_Accept() >= 0) {
        printf("Content-type: text/plain\r\n\r\n"
               "Request number %d\n", ++count);
        for (i = 0; i < MAX_VARS; ++i) {
            v = getenv(vars[i]);
            if (v == NULL)
                printf("%s: \n", vars[i]);
            else
                printf("%s: %s\n", vars[i], v);
        }
    }
    return 0;
}

We compile this source code as follows:

$ gcc -o echo.fcgi echo.c -lfcgi

and store the resulting binary in /home/user/fcgiws/echo.fcgi, as specified in the lighttpd configuration above.

To deploy this web service, all we need to do now is restart lighttpd server. On Ubuntu, you run the following:

$ sudo service lighttpd restart
  * Stopping web server lighttpd [ OK ]
  * Starting web server lighttpd [ OK ]

If you look at the log file at server.errorlog, you will see the following:

$ cat /var/log/lighttpd/error.log
  2014-12-18 17:07:57: (log.c.166) server started
  2014-12-18 17:07:57: (mod_fastcgi.c.1367) --- fastcgi spawning local
          proc: /home/user/fcgiws/echo.fcgi
          port: 0
          socket /tmp/echo.fcgi.socket
          max-procs: 1
  2014-12-18 17:07:57: (mod_fastcgi.c.1391) --- fastcgi spawning
          port: 0
          socket /tmp/echo.fcgi.socket
          current: 0 / 1

Testing the web service

We shall test the web service using two different user-agents. One will be the browser, say Google Chrome, and the other is the wget command. The URL to target, based on the lighttpd configuration above, is http://localhost/echo. So, if you load a URL say

http://localhost/echo?first=Apple&second=Orange

on the browser, you will get a page that looks something like:

Request number 3
DOCUMENT_ROOT: /var/www
GATEWAY_INTERFACE: CGI/1.1
HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
HTTP_ACCEPT_ENCODING: gzip, deflate, sdch
HTTP_ACCEPT_LANGUAGE: en-GB,en-US;q=0.8,en;q=0.6,af;q=0.4
HTTP_CACHE_CONTROL:
HTTP_CONNECTION: keep-alive
HTTP_HOST: localhost
HTTP_PRAGMA:
HTTP_RANGE:
HTTP_REFERER:
HTTP_TE:
HTTP_USER_AGENT: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
HTTP_X_FORWARDED_FOR:
PATH:
QUERY_STRING: first=Apple&second=Orange
REMOTE_ADDR: 127.0.0.1
REMOTE_HOST:
REMOTE_PORT: 39225
REQUEST_METHOD: GET
REQUEST_URI: /echo?first=Apple&second=Orange
SCRIPT_FILENAME: /var/www/echo
SCRIPT_NAME: /echo
SERVER_ADDR: 127.0.0.1
SERVER_ADMIN:
SERVER_NAME: localhost
SERVER_PORT: 80
SERVER_PROTOCOL: HTTP/1.1
SERVER_SIGNATURE:
SERVER_SOFTWARE: lighttpd/1.4.28

If you load the same URL using wget, you will get something like:

$ wget --output-document echo.html "http://localhost/echo?first=Apple&second=Orange"
$ cat echo.html
  Request number 4 on host <i>GET</i>
  DOCUMENT_ROOT: /var/www
  GATEWAY_INTERFACE: CGI/1.1
  HTTP_ACCEPT: */*
  HTTP_ACCEPT_ENCODING:
  HTTP_ACCEPT_LANGUAGE:
  HTTP_CACHE_CONTROL:
  HTTP_CONNECTION: Keep-Alive
  HTTP_HOST: localhost
  HTTP_PRAGMA:
  HTTP_RANGE:
  HTTP_REFERER:
  HTTP_TE:
  HTTP_USER_AGENT: Wget/1.13.4 (linux-gnu)
  HTTP_X_FORWARDED_FOR:
  PATH:
  QUERY_STRING: first=Apple&second=Orange
  REMOTE_ADDR: 127.0.0.1
  REMOTE_HOST:
  REMOTE_PORT: 39399
  REQUEST_METHOD: GET
  REQUEST_URI: /echo?first=Apple&second=Orange
  SCRIPT_FILENAME: /var/www/echo
  SCRIPT_NAME: /echo
  SERVER_ADDR: 127.0.0.1
  SERVER_ADMIN:
  SERVER_NAME: localhost
  SERVER_PORT: 80
  SERVER_PROTOCOL: HTTP/1.1
  SERVER_SIGNATURE:
  SERVER_SOFTWARE: lighttpd/1.4.28

Further details on some of the CGI environment variables are available here and here. There are several other CGI tutorials on the web, which provide further details. I do hope, however, that the working example described in this note will help you get started with some exciting experimentation. It will be interesting to profile and compare production-level FastCGI web services against those that use other web service frameworks.