In the Observer design pattern, a subject holds a list of interested parties – the observers – which it will notify about changes in status. Simply put, it’s a form of subscription, and this design comes up in all sorts of places (which is one of the definitions of the term ‘design pattern‘). It’s well suited for handling asynchronous events, like user interaction in a GUI, sensor information, and so on.

There is, however, often a need to re-synchronise asynchronous events. For instance, you might keep the latest status update until it’s actually needed for display, storage or some calculation. By doing this, you disregard the asynchronous nature of its source, and treat it as just another variable, as if it had been read from the subject right then. In other words, you synchronise a status from the past with the present. Sometimes, though, you don’t want the last value, but the next, which is a bit more complex, as it requires you to wait for the future to happen before we can say it’s the present.

In this article, we will write a simple multi-threaded example implementation of the Observer pattern, and show how to re-synchronise a past event to look current. Then we’ll demonstrate a technique to treat future events like they’re current, too.

Thread safety

If, as is often the case, the observed subject and the part of the system that manages the observers run in different threads, we need to make sure that they can co-exist in a friendly manner. Specifically, we must make sure that adding or removing an observer – activities that takes place outside the observed thread – does not interfere with the reporting of events.

In other words, accessing the list of observers is a critical section of the code, which may not be interrupted. Since this is a quite common situation, operating systems that support multi-threading also provide tools to handle it. On Windows, this is done with a CRITICAL_SECTION object.

(If you use MFC, there is an eponymous wrapper for it. However, the implementations of the MFC synchronisation objects have been badly flawed in the past, and I believe they still are.)

Checking the documentation, we see that there are functions available to create and destroy CRITICAL_SECTION, and to enter and leave a locked state. We also see that it can be entered and left recursively, as long as it’s the same thread. Knowing all this, we can write a C++ class to manage it.

Why not use the Windows API directly? The same reason for almost all C++ wrappers of OS objects – lifetime management. Instead of having to remember to call DeleteCriticalSection everywhere it might be needed, we can do that in the destructor, as per the Resource Acquisition Is Initialization idiom.

// Need CRITICAL_SECTION declaration
#include <Windows.h>

// Class wrapping Win32 CRITICAL_SECTION
class CriticalSection
{
public:
  // Constructor
  CriticalSection()
  { 
    ::InitializeCriticalSection(&cs_); 
  }

  // Destructor
  ~CriticalSection()
  { 
    ::DeleteCriticalSection(&cs_); 
  }
  
  // Enter critical section
  void Enter()
  { 
    ::EnterCriticalSection(&cs_); 
  }

  // Leave critical section
  void Leave()
  { 
    ::LeaveCriticalSection(&cs_);
  }

private:
  // Hide copy operations
  CriticalSection(const CriticalSection&);
  CriticalSection& operator=(const CriticalSection&);
  
  // Data member
  CRITICAL_SECTION cs_;
};

Quite simple, really, with little overhead. And because every Enter must be matched by a Leave, the sensible thing to do is to write a RAII wrapper for that, too. If we don’t, odds are that at some point, we’ll alter the code using the CriticalSection and introduce a new exit point, via a function return or exception, which won’t get the Leave function called. A RAII wrapper helps code robustness.

// RAII Critical section lock
class CSLock
{
public:
  // Constructor
  CSLock(CriticalSection& section)
  : section_(section) 
  { 
    section_.Enter(); 
  }
  
  // Destructor
  ~CSLock()
  { 
    section_.Leave(); 
  }
    
private:
  // Hide copy operations
  CSLock(const CSLock&);
  CSLock& operator=(const CSLock&);
    
  // Data member
  CriticalSection& section_;
};

This automates the locking, so that we only need declare a CSLock at the beginning of the scope we wish to lock for interruption, and will automatically unlock as we leave the scope and the destructor is called. We’ll see examples of how these are used below.

Something to see

Now that we have the tools to support a multi-threaded application, let’s write a simple Observer system. This requires something to observe, and something to do the observing. For this example, we’ll make a Subject class, which declares an internal, abstract Subject::Observer class, from which we’ll derive our observers. The Subject notification in this example will send an integer to the observers.

#include <set>
#include "CSLock.h"

// Subject to be observed
class Subject
{
public:
  // Abstract observer
  class Observer
  {
    // The subject we're observing
    Subject* subject_;
    // The subject is a friend, so it can help manage our relationship
    friend class Subject;

    // Assign a subject we've been registered with
    void SetSubject(Subject* subject)
    {
      // Is it the one we have already?
      if (subject != subject_)
      {
        // Unregister from current subject, if any
        if (0 != subject_)
        {
          subject_->Unregister(*this);
        }
        // Remember new subject
        subject_ = subject;
      }
    }

    // Kicked out by the subject
    void ReleaseFromSubject(Subject* subject)
    {
      if (subject == subject_)
      {
        subject_ = 0;
      }
    }
  protected:
    // Constructor only available to derived classes
    Observer()
    : subject_(0)
    {}

    // Derived classes decide what to do with notifications
    // (Still available to Subject class, as it's a friend)
    virtual void Notify(int) = 0;

  public:
    // Base classes need virtual destructor
    virtual ~Observer()
    {
      SetSubject(0);
    }
  };
  // End of internal class definition

  // Destructor
  ~Subject()
  {
    // Lock, as we're using the set of observers
    CSLock lock(criticalsection_);
    // Release all observers
    for (std::set<Observer*>::iterator i = observers_.begin(); 
         i != observers_.end(); ++i)
    {
      (*i)->ReleaseFromSubject(this);
    }
  }

  // Add observer
  void Register(Observer& observer)
  {
    // Lock, as we're manipulating the set of observers
    CSLock lock(criticalsection_);
    // Add to the set of observers
    observers_.insert(&observer);
    // Let it know we've accepted the registration
    observer.SetSubject(this);
  }
  
  // Remove observer
  void Unregister(Observer& observer)
  {
    // Lock, as we're manipulating the set of observers
    CSLock lock(criticalsection_);
    // Remove from the set of observers
    observers_.erase(&observer);
    // Let it know we've accepted the unregistration
    observer.ReleaseFromSubject(this);
  }

  // Notify observers about new data
  void Notify(int val) const
  {
    // Lock, as we're using the set of observers
    CSLock lock(criticalsection_);
    // Notify all
    for (std::set<Observer*>::const_iterator i = observers_.begin(); 
         i != observers_.end(); ++i)
    {
      (*i)->Notify(val);
    }
  }

  // Check how many observers we have
  size_t ObserverCount() const
  {
    return observers_.size();
  }

private:
  // The registered observers
  std::set<Observer*> observers_;
  // A critical section to guard the observers
  mutable CriticalSection criticalsection_;
};

The first thing to note here is that the Subject and Observer are tightly coupled, which somewhat paradoxically is to help de-couple the derived observers from the Subject. The logic and responsibility of maintaining the relationship is kept private, thanks to the friendship between the Subject and Observer, so that derived classes can’t affect it. This tight coupling is also the reason for making the Observer an internal class, to emphasise this is not any old observer, but one for this particular Subject.

Another thing worth noting is that just as the Subject::Observer class leaves the actual handling of a Notify call to a derived class, the Subject class here isn’t concerned with the generation of values to notify observers with. That’s for someone else, this Subject only handles its observers and getting notifications out to them. (Indeed, it would be a relatively trivial task to make the notification type (int in this example) a template type, and make this a generic and re-usable Observer pattern implementation. To do so is left as an exercise to the reader. Just mind whether you notify by value, reference, or pointer.)

A final point worth making is that the CriticalSection is declared to be mutable. The reason for this is that it’s only altered during a function call, by the CSLock, but at the end of the function call it will have been restored to its previous state. By indicating it’s mutable, we can make the Notify function const.

So, let’s put it all together, with a custom observer that saves the latest value, a function to produce values, a thread, and a complete program.

#include <iostream>

class PastObserver : public Subject::Observer
{
  // Last observed value
  int value_;
  // A critical section to guard the value
  mutable CriticalSection criticalsection_;

public:
  // Constructor, 
  PastObserver()
  : value_(0)
  {}

  // Access last value
  int GetLastValue() const
  {
    // Lock to prevent the value being modified
    CSLock lock(criticalsection_);
    return value_;
  }

  // Function called by observed Subject
  virtual void Notify(int value)
  {
    {
      // Lock to prevent the value being read while we're assigning
      CSLock lock(criticalsection_);
      // Store the value
      value_ = value;
    }
    // Print it out
    std::cout << "PastObserver notified: " << value_ << std::endl;
  }
};

// A thread function to generate values
// Takes a Subject* as thread parameter
DWORD WINAPI ValueFunction(void* pParam)
{
  Subject* subject = (Subject*)pParam;

  std::cout << "Thread started with " << 
    subject->ObserverCount() << " observers" << std::endl;

  // Put a seed value in for the random number generator
  int val = (int)time(0);
  // Run until there are no more observers
  while (0 < subject->ObserverCount())
  {
    // Get a random value to report
    srand(val);
    val = rand();
    // Report it
    subject->Notify(val);
    // Take a little break
    Sleep(100 * (val & 0x7));
  }
  std::cout << "Thread ended" << std::endl;
  return 0;
}

void main()
{
  // Create subject and observer
  Subject subject;
  PastObserver observer;
  subject.Register(observer);
  
  // Start the thread
  CreateThread(NULL, NULL, &ValueFunction, &subject, NULL, NULL);

  // Let it work for a bit
  Sleep(1000);

  // Close down and report
  subject.Unregister(observer);
  std::cout << "Last value: " << observer.GetLastValue() << std::endl;

  // Wait for thread to terminate
  ::WaitForSingleObject(thread, INFINITE);
} 

And that’s it. A reasonably small and clear illustration of how the Observer pattern works. In this example, we synchronise with the past, by reading the last value. Now, let’s synchronise with the future!

Reading the future

So how do you read the future? Well, obviously, our observer has to wait for it to happen, so we’ll need another synchronisation object: the Event. This is a boolean object which can be set or reset, and waited for with WaitForSingleObject. As it turns out, we’ll need two of those – one to indicate we’re waiting for data, and one to indicate we’ve received it.

class FutureObserver : public Subject::Observer
{
  // Observed value
  int value_;

  // Events 
  HANDLE waiting_;
  HANDLE newValue_;

public:
  // Constructor
  FutureObserver()
  : value_(0),
    waiting_(0),
    newValue_(0)
  {
    // No security attributes, automatic reset, initially reset, no name
    waiting_ = ::CreateEvent(0, 0, 0, 0);
    newValue_ = ::CreateEvent(0, 0, 0, 0);
  }

  // Destructor
  ~FutureObserver()
  {
    CloseHandle(waiting_);
    CloseHandle(newValue_);
  }

  // Wait for next value
  int GetNextValue() const
  {
    // Indicate we're waiting
    SetEvent(waiting_);

    // Wait for a new value
    if (WAIT_OBJECT_0 == ::WaitForSingleObject(newValue_, INFINITE))
    {
      // Success
      return value_;
    }
    else
    {
      // Failures, could be WAIT_FAILED or WAIT_TIMEOUT
      throw std::exception("Failed waiting for next value");
    }
  }

  // Function called by observed Subject
  virtual void Notify(int value)
  {
    // Print it out
    std::cout << "FutureObserver notified: " << value << std::endl;
    // Check to see if we're waiting
    if (WAIT_OBJECT_0 == ::WaitForSingleObject(waiting_, 0))
    {
      // We were waiting, so keep the value...
      value_ = value;
      // ... and flag we have it
      ::SetEvent(newValue_);
    }
  }
};

This observer is a bit more complex, as it has to manage the two Event objects, but the principle is simple enough. In the GetNextValue() function, we set one event, and wait for the other. The next time Notify() is called, it will see the waiting_ flag is set, so it will store the value and signal that newValue_ is ready. The events are created to reset automatically, as soon as a WaitForSingleObject call is successful (eg when the event it is waiting for has been set).

The GetNextValue() function waits infinitely here – it will not continue until it’s found a value – and so the exception should never happen, unless the FutureObserver has been deleted in another thread. If you’d prefer a timeout, just overload the GetNextValue() function:

  // Access next value, with timeout and success indicator
  int GetNextValue(DWORD millisecondTimeout, bool& timedOut) const
  {
    // Indicate we're waiting
    SetEvent(waiting_);

    // Wait for a new value
    switch (::WaitForSingleObject(newValue_, millisecondTimeout))
    {
    case WAIT_OBJECT_0:
      // Success
      timedOut = false;
      return value_;
    case WAIT_TIMEOUT:
      timedOut = true;
      return 0;
    default:
      // WAIT_FAILED
      throw std::exception("Failed waiting for next value");
    }
  }

The Notify() function, in contrast, doesn’t wait at all. When WaitForSingleObject is called with a timeout of zero milliseconds, it returns immediately, so we have to check the return value to see if we were successful. This means we’re not holding up the Subject::Nofify() more than necessary.

Finally, let’s put it all together:

void main()
{
  // Create subject and observers
  Subject subject;
  PastObserver past;
  subject.Register(past);
  FutureObserver future;
  subject.Register(future);
  
  // Start the thread
  HANDLE thread = CreateThread(NULL, NULL, &ValueFunction, &subject, NULL, NULL);

  // Let it work for a bit
  Sleep(1000);

  // Get the last and next value, twice
  std::cout << "Last value: " << past.GetLastValue() << std::endl;
  std::cout << "Next value: " << future.GetNextValue() << std::endl;
  std::cout << "Last value: " << past.GetLastValue() << std::endl;
  std::cout << "Next value: " << future.GetNextValue() << std::endl;

  // Close down
  subject.Unregister(past);
  subject.Unregister(future);

  // Wait for thread time to terminate
  ::WaitForSingleObject(thread, INFINITE);
} 

As always, if you found this interesting or useful, or have suggestions for improvements, please let me know.

// Needs Windows constant and type definitions
#include <windows.h>

// Create a string with last error message
std::string GetLastErrorStdStr()
{
  DWORD error = GetLastError();
  if (error)
  {
    LPVOID lpMsgBuf;
    DWORD bufLen = FormatMessage(
        FORMAT_MESSAGE_ALLOCATE_BUFFER | 
        FORMAT_MESSAGE_FROM_SYSTEM |
        FORMAT_MESSAGE_IGNORE_INSERTS,
        NULL,
        error,
        MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
        (LPTSTR) &lpMsgBuf,
        0, NULL );
    if (bufLen)
    {
      LPCSTR lpMsgStr = (LPCSTR)lpMsgBuf;
      std::string result(lpMsgStr, lpMsgStr+bufLen);
      
      LocalFree(lpMsgBuf);

      return result;
    }
  }
  return std::string();
}

This function retrieves the last error code, if any, and gets the text message associated with it, which is then converted to a standard string and returned. The main benefits of using this function is that it saves you from having to remember the syntax of FormatMessage, and that the memory used is tidied up.

Note that the FORMAT_MESSAGE_FROM_SYSTEM flag means only system error messages will be given. If you want to include error messages from your own modules, you’ll need to add the FORMAT_MESSAGE_FROM_HMODULE flag, and provide the handle to the module. See the FormatMessage documentation for details.

One of the more curious omissions from the C++ standard library is a string splitter, e.g. a function that can take a string and split it up into its constituent parts, or tokens, based on some delimiter. There is one in other popular languages ((C# – String.Split, Java – String.split, Python – string.split etc), but C++ programmers are left to roll their own, or use one from a third-party library like the boost::tokenizer (or the one I presented in Splitting strings).

There are many ways of going this; the Stack Overflow question How do I tokenize a string in C++? has 23 answers at the time of writing, and those contain 20 different solutions (boost::tokenizer and strtok are suggested multiple times).

The strtok recommendations, however, all have comments pointing out the problems with this function – it’s destructive, and not reentrant (it can’t be nested or run in parallell on multiple strings). As functions go, strtok has a rather poor reputation – there’s even a popular reentrant version, strtok_r, available in many C library implementations, though it’s not a standard function.

So it’s a good thing that there are so many other ways of splitting strings, isn’t it? Well, yes, but the clever thing with strtok, which you don’t get from any of the string splitters, not even the flexibility-obsessed Boost, is that you can change the token separator as you go along. The splitters often give the option to provide a selection of possible delimiters like this:

  char punct = " ,.?!;:-";
  std::string text = "Silly, right? This is - obviously - an example: Boo!";
  
  std::vector<std::string> tokens;
  std::vector<char> separators(punct, punct + strlen(punct));
  
  tokenise_string(text, separators, tokens);
  // tokens now contains the eight words of the string

But what if there are characters that are both legal inside some tokens, and separators for others? What if you have to parse strings with the format [Last name],[First name] – [Profession] – [Age] like this table:

Bradshawe, Adam - Colonel, retired - 73
Burton-West,Jenny - Surgeon - 37
Smith,Ben - Taxi driver - 56

While a string splitter would stumble over this – since both space, hyphen and comma can be both delimiters and valid content – old strtok doesn’t even raise an eyebrow:

  const char* table = "Jones, Adam - Colonel, retired - 73\n"
    "Burton-West,Jenny - Surgeon - 37\n"
    "Smith,Ben - Taxi driver - 56";
  char* changeable = (char*) malloc(strlen(table)); // cast not needed in C
  strcpy(changeable, table);

  char *last, *first, *profession, *age;
  // Start it off with the first search
  last = strtok(changeable, " ,");
  while (last)
  {
    first = strtok(NULL, " -");
    profession = strtok(NULL, "-");
    // Only snag - this probably has a trailing space we need to trim
    int proflen = strlen(profession);
    if (profession[proflen - 1] == ' ')
      profession[proflen - 1] = '';
      
    age = strtok(NULL, " -\n");
    // Have whole row, take care of data
    // ...
    // Start next row, if any
    last = strtok(NULL, " ,");
  }
  free(changeable);

In other words, strtok offers unique functionality not found in the string splitters. So let’s design a C++ version that is efficient, non-destructive, and reentrant. Thanks to the object-orientation support of C++, we can let each tokeniser have a const reference to the string we’re tokenising. This gives us both reentrancy and non-destructiveness. And while we’re at it, let’s have a flag to decide whether we should include empty tokens or not – something quite useful that’s missing from strtok.

  class string_tokeniser
  {
    // The string we're searching in
    const std::string& source_;
    // Flag indicating whether to include empty strings
    bool empty_;
    // Current location in string
    std::string::size_type current_;
    // Length of current string
    std::string::difference_type length_;
    // Location to start next search
    std::string::size_type next_;
    
  public:
    // Constructor, setting the string to work on
    string_tokeniser(const std::string& source, bool empty = false);
    ...

We’ll also need variables to keep track of where we are, where to start looking for next token, and so on, which I’ve also included above.

Now, what do we want to do with this? Well, we actually want to do two distinct tasks – advance to the next token, and extract a token. In strtok, those are done at the same step, but since we’re not constrained by the limitations of C, it’s better to keep it tidy. Like I did in my string splitter, I’ll overload the advancing function to allow the user to give a single character, a selection of characters, or a whole string as a delimiter.

    ...
    // Advance to next token, by given character separator
    bool next(const std::string::value_type& separator);

    // Advance to next token, by any of given character separators
    bool next(const std::vector<char>& separators);

    // Advance to next token, by given string separator
    bool next(const std::string& separator);
    ...

These all return true if a new token was found in the string. We’ll also need a way of accessing the tokens found, and some housekeeping:

    ...
    // Get current token, if any, safely
    bool get_token(std::string& token) const;

    // Get current token, if any, otherwise an empty token
    std::string get_token() const;

    // Reset search
    void reset();

    // Check token availability
    bool has_token() const;

    // Check if search is at end
    bool at_end() const;

    // Get source string
    const std::string& get_source() const;
  };

That’s the interface, let’s do some implementation. The way this will work is that we keep hold of a current location and token length, which are used to retrieve the token using std::string::substr, while also keeping the next location in which to start the search for a token. This will have to be next_ = current_ + length_ + length of delimiter, so the next search does not pick up the last delimiter.

When the string_tokeniser is first created, we have no search results, so need to initialise appropriately. The same values are set on a reset, and used to get the current token and check status:

  // Constructor
  string_tokeniser::string_tokeniser(const std::string& source, bool empty /*= false*/)
    : source_(source)
    , current_(std::string::npos)
    , length_(0)
    , next_(source.empty() ? std::string::npos : 0)
    , empty_(empty)
  {}

  // Reset search so it can be restarted
  void string_tokeniser::reset()
  {
    current_ = std::string::npos;
    next_ = source_.empty() ? std::string::npos : 0;
    length_ = 0;
  }

  // Return true if there is a current token
  bool string_tokeniser::has_token() const
  {
    // Not worried about length here, as it might be an empty token
    return (std::string::npos != current_);
  }

  // Return true if no further searches can be done
  bool string_tokeniser::at_end() const
  { 
    return (std::string::npos == next_);
  }

  // Get source string
  const std::string& string_tokeniser::get_source() const
  {
    return source_;
  }

  // Get current token, if any, safely
  bool string_tokeniser::get_token(std::string& token) const
  {
    if (!has_token())
      return false;
    if (0 == length_)
      token.clear();
    else
      token = source_.substr(current_, length_);
    return true;
  }

  // Get current token, if any, otherwise an empty token
  std::string string_tokeniser::get_token() const
  {
    if (!has_token() || (0 == length_))
      return std::string();
    return source_.substr(current_, length_);
  }

Right, that just leaves the implementation of the key function: next(). Since there are three overloads, with almost identical implementation, the sensible thing is to break out most of the common stuff into a helper function. Unfortunately, we can’t easily do that, since if we do not care about empty tokens, we have to recurse and try to find the next, in the case of repeated delimiters, which means it will have to be aware of which overload to chose. Instead, we’ll break out the common handling into a template function:

In class declaration:
    ...
    // Helper - handle the result of a search, advancing to prepare for next
    template <typename T>
    bool handle_next(size_t advance, const T& separator);
  public:
    // Constructor, setting the string to work on
    ...

Implementation:
  // Advance to next token
  bool string_tokeniser::next(const std::string::value_type& separator)
  {
    // Store the start
    current_ = next_;
    if (at_end())
      return false;
    // Find next
    next_ = source_.find(separator, current_);
    // Deal with result of search
    return handle_next(1, separator);
  }

  // Advance to next token
  bool string_tokeniser::next(const std::string& separator)
  {
    // Store the start
    current_ = next_;
    if (at_end())
      return false;
    // Find next
    next_ = source_.find(separator, current_);
    // Deal with result of search
    return handle_next(separator.size(), separator);
  }

  // Advance to next token
  bool string_tokeniser::next(const std::vector<char>& separators)
  {
    // Store the start
    current_ = next_;
    if (at_end())
      return false;
    // Find next
    next_ = source_.find_first_of(&separators[0], current_, separators.size());
    // Deal with result of search
    return handle_next(1, separators);
  }

  // Handle the result of a search, advancing to prepare for next search
  template <typename T>
  bool string_tokeniser::handle_next(size_t advance, const T& separator)
  {
    if (std::string::npos == next_)
    {
      // Separator not found, but there might still be data, at the end 
      length_ = source_.size() - current_;
    }
    else
    {
      // Store the length of the current token
      length_ = next_ - current_;
      // and move next starting point to beyond the one we found
      next_ += advance;
      // In the case of double separators (e.g. | in "a|b||d"), this gives an 
      // empty token. If empties aren't accepted, we'll recurse
      if ((0 == length_) && !empty_)
      {
        return next(separator);
      }
    }
    // Do we have a token?
    if (0 < length_)
      return true;
    // Even if empties are accepted, dismiss an empty token at the end of a 
    // string (e.g. "a|b|" gives "a" and "b" only)
    if (!empty_ || (std::string::npos == next_))
    {
      // Invalidate current, so extraction isn't valid
      current_ = next_;
      return false;
    }
    return true;    
  }

There, all done. And because we can use both single characters, a selection of characters, and strings as delimiters, the equivalent of the strtok example avoids the need to trim spaces:

  std::string table = "Jones, Adam - Colonel, retired - 73\n"
    "Burton-West,Jenny - Surgeon - 37\n"
    "Smith,Ben - Taxi driver - 56";

  std::string last, first, profession, age;
  // Prepare delimiters to use
  std::vector<char> comma_space;
  comma_space.push_back(' ');
  comma_space.push_back(',');
  std::string sp_dash_sp(" - ");
  char endl('\n');

  string_tokeniser tok(table);
  while (!tok.at_end())
  {
    tok.next(comma_space);
    tok.get_token(last);
    tok.next(sp_dash_sp);
    tok.get_token(first);
    tok.next(sp_dash_sp);
    tok.get_token(profession);
    tok.next(endl);
    tok.get_token(age);
  }

Because it is non-destructive, it is by necessity less efficient than strtok, since the tokens have to be copied. On the other hand, if you have to copy the const char* to a char* buffer to use strtok, maybe the efficiency loss isn’t that bad.

As always, if you found this interesting or useful, or have suggestions for improvements, please let me know.

Update: Jens Ayton has informed me that C11 introduced strtok_s, which is even safer, but not, I believe, incorporated into C++11 .

There is a quote that goes “Standards are great! Everyone should have one.” or something along those lines. (Somewhat ironically, this quote, too, has many different variations, and has many attributions. The earliest I’ve found attributes it to George Morrow in InfoWorld 21 Oct 1985).

A case in point is the base64 encoding. Put simply, it’s a method of encoding an array of 8-bit bytes using an alphabet consisting of 64 different printable characters from the ASCII character set. This is done by taking three 8-bit bytes of source data, arranging them into a 24-bit word, and converting that into four 6-bit characters that maps onto the 64-character alphabet (since 6 bits is 0-63).

The original implementation was for privacy-enhanced e-mail (RFC 1421), then altered slightly for MIME (RFC 2045), and again in its own standard (RFC 4648).

When I was looking at base64, I was interested in three different varieties or flavours, namely the MIME version, the (per RFC 4648) standard base64, and base64url. These differ in how they handle line breaks and other illegal characters, what characters are used in the 64-character alphabet, and the use of padding at the end to make up an even triplet of bytes.

That’s three mostly similar but slightly different algorithms, so how to design an implementation? A number of designs are possible, of course, like:

1. three copy-pasted and tweaked sets of functions
2. hugely parameterised functions where every variation in algorithm or data can be altered in the call
3. an inheritance tree, where virtual overridden functions alter behaviour

I chose to use a design that in my opinion takes the best from all of those, with none of the disadvantages.

The Wikipedia article has a handy table giving a summary of the differences between the variants, which gives eight possible differences of data or algorithm. I’ve elected to ignore the last of those, which is the addition of a checksum only used for OpenPGP (RFC 4880), since that is easily added outside the base64 coding. The sixth difference on Wikipedia’s list – line separator – can also be ignored since it can be inferred from the maximum line length. The remaining areas of difference are:

• Character for index 62
• Character for index 63
• Character to use for padding
• Whether fixed line length is used
• Maximum line length
• Handling of illegal characters

Fortunately, these are all integer types (bool and char are both integer types), so can be used as template parameters. In other words, I can define a type to hold all possible variants:

template<
  char Tchar62         // Character for index 62 in alphabet
  char Tchar63         // Character for index 63 in alphabet
  char TcharPad        // Character to use for padding, or 0 if 
                       // padding is not used
  bool TfixLineLength  // false if line length is fixed. If true, 
                       // maxLineLength is used
  int  TmaxLineLength  // Maximum (or fixed) line length. 0 if not 
                       // used. If used, CR+LF is used as linebreak
  bool TignoreIllegal> // if false, discards characters not in 
                       // alphabet; if true throws error on finding 
                       // illegal character
struct base64_variants
{
  // This type only has type information accessors
  // Get character for index 62 in alphabet
  static char char62()  
    { return Tchar62; } 
    
  // Get character for index 63 in alphabet
  static char char63()  
    { return Tchar63; }          
    
  // Get padding character, if any
  static char charPad() 
    { return TcharPad;  }
    
  // Check if padding is used
  static bool pad()     
    { return TcharPad > 0;  }
    
  // Check if line length is fixed
  static bool fixLineLength() 
    { return TfixLineLength;  }
    
  // Get maximum line length
  static int  maxLineLength() 
    { return TmaxLineLength;  }
    
  // Check if lines should be broken
  static bool lineBreaks()    
    { return TmaxLineLength>0;  }
    
  // Check if illegal characters may be ignored
  static bool ignoreIllegal() 
    { return TignoreIllegal;  }
};

What is the benefit of this? Well, it lets us express distinct sets of parameters to correspond to particular standards, like this:

// MIME variety of base64 (RFC 2045) 
typedef base64_variants<'+', '/', '=', false, 76, true> base64MIME;

// Plain standard (RFC 4648) base64
typedef base64_variants<'+', '/', '=', false, 0, false> base64;

// URL variety of base64 (RFC 4648)
typedef base64_variants<'-', '_', 0, false, 0, false> base64url;

This means that there is no risk of mixing up parameters in function calls – once a set is defined (and tested and verified to be correct) that can be used everywhere. The actual coding functions then get a very simple interface, with an input, an output, and variant type as only parameters:

// Encode C-style array of bytes
template<typename T>
void bytes_to_base64(const unsigned char* data, 
        size_t length, 
        std::string& str);

// Encode byte vector
template<typename T>
void bytes_to_base64(const std::vector<unsigned char>& data, 
        std::string& str);

// Decode to byte vector        
template<typename T>
void base64_to_bytes(const std::string& str, 
        std::vector<unsigned char>& data);

And this is how the use of these would look in code:

// Using test vectors from RFC 4648
const char* src = "fooba";
std::vector<unsigned char> data(src, src+5);
std::string result;

bytes_to_base64<base64>(data, result);

assert(result == "Zm9vYmE=");

base64_to_bytes<base64>(result, data);

result = std::string(data.begin(), data.end());
assert(result == "fooba");

Below is a short extract of the implementation, illustrating how the variant specification is used:

// Encode
template<typename T>
void bytes_to_base64(const unsigned char* data, size_t length, 
  std::string& str)
{
  str.clear();
  // Calculate maximum expected size (bytes, padding, line breaks) 
  str.reserve((length * 4 ) / 3 + 3 + 
    (T::lineBreaks() ? 2 * length / T::maxLineLength() : 0) );

(I won’t post the whole implementation here. While it’s only 200 airy and thoroughly commented lines, it’s better to give you links to all the files, which I do at the beginning of this post.)

Now, the astute reader will note that I only declared the coding functions earlier, and talked about the implementation as if it was separate. That’s the normal way of doing things, but that won’t work with templates, right?

Here’s the thing with template functions and classes: they are not just the one thing. A normal, non-template function is one single thing, fully defined once and once only. Therefore, it can be compiled in one compilation unit, and then linked to. It can be declared elsewhere, and that declaration is essentially a promise that somewhere this thing is defined, which is all the compiler cares about.

A template function is effectively a new function for each template parameter (or combination of parameters) it is used with. (As far as the compiler is concerned, a template class is just a way of saying that all member functions have the same set of template parameters.) To the compiler, there’s no such thing as a “template function”. There is only “template function with these template parameters”.

This means that there can’t be a single compiled variant that can be linked to, only specialisations that use this or that set of template parameters. Even if only one variant, only one specialisation, is used in your project, the compiler can’t know that.

So instead, the compiler compiles these inline; it’s effectively replacing each call to a template function with a copy of that function, in which the template parameters are those used in that particular call.

In other words, C++ templates are just a way of bullying the compiler into doing the copy-paste programming you are ashamed of doing yourself.

As it happens, though, that also provides the solution to separationg definitions and declarations, provided you know what template parameters you’ll be using. All you need to do is declare the function with the parameters you want, in the same compilation unit as the template function definition.

Here’s a simple example:

-------------------------------------
-- In my_template.h file
... 
template <typename T>
T my_temp_func(const T& t);

-------------------------------------
-- In my_template.cpp file

#include "my_template.h"

// Definition
template <typename T>
T my_temp_func(const T& t)
{
  return t;
}

// Instantiation declaration
int my_temp_func<int>(const int& t);

-------------------------------------
-- In using_my_template.cpp
#include "my_template.h"
...
int i = my_temp_func<int>(4); // works
string s = my_temp_func<string>("Abob"); // link error

In the example above, the declaration in the last line of my_template.cpp tells the compiler there’ll be a variant of the template function that uses int as template parameter. Okay, says the compiler, I’ll put an inline copy there. Since the generic definition is right there in the same compilation unit (ie my_template.cpp), this is something the compiler can do – it has all the information it needs.

The result of that is that in the compiled file (probably called my_template.obj) there is now a function that has the signature int my_temp_func(const int& t). This is a fully defined specialisation of a template function, so to the linker it looks just like a normal function.

However, the linker won’t be able to find a string specialisation, so this will generate a linker error.

This illustrates both how to use this trick, and its limitation. It only works if you list all specialisations you are going to use, which makes it unfeasible for generic libraries.

In this case, though, it’s ideal. I have my three variants of base64 defined – base64MIME, base64 and base64url – and those are the only one I’ll need.

Actually, I might as well add a definition for the original variant:

// PEM variety of base64 (RFC 1421)
typedef base64_variants<'+', '/', '=', true, 64, false> base64PEM;

So I have my four variations defined, and they are the only variations of base64 I’m interested in, so I’ll just have to declare them in the same .cpp file as the function definitions are in:

template void bytes_to_base64<base64>(const unsigned char* data, 
                    size_t length, std::string& str);
template void bytes_to_base64<base64>(const std::vector<unsigned char>& data, 
                    std::string& str);
template void base64_to_bytes<base64>(const std::string& str, 
                    std::vector<unsigned char>& data);

template void bytes_to_base64<base64MIME>(const unsigned char* data,
                    size_t length, std::string& str);
...                  

This lets the linker find compiled varieties for all those base64 definitions.

Should you want to implement a slightly different base64 coder, you could use the code I’ve written. It wouldn’t be enough to declare a new definition type, but you would also have to add a declaration using that type to the .cpp file. But the source code is both open and free, so help yourself.

(I should note that like with so much else, base64 is something there are lots and lots of implementations of available on the net, but most of the ones I’ve found tended to be very lax and lack strict checking of syntactical correctness, or implement just one flavour. Hence, writing my own.)

As always, if you found this interesting or useful, or have suggestions for improvements, please let me know.

To start with, how do you identify whitespace? Let’s have a look at some different approaches (all of which I’ve seen in the wild):

// Simple
bool iswhitespace1(char c)
{
  // Is it  space   or    tab      or    return   or    newline?
  return (c == ' ') || (c == '\t') || (c == '\r') || (c == '\n');
}
// Cute attempt at cleverness
bool iswhitespace2(char c)
{
  // Is it one of the whitespace characters?
  static const std::string spaces(" \t\r\n");
  return (std::string::npos != spaces.find(c));
}
// Probably ok, for English at least
bool iswhitespace3(char c)
{
  // Using C function, from <cctype>
  return ::isspace(c);
}
// As above, but standard C++ instead of standard C
bool iswhitespace4(char c)
{
  // Using current locale, and std function from <locale>
  static const std::locale loc;
  return std::isspace(c, loc);
}

If we were to run through these four functions with values of c from 0 to 255, the first two would produce the same result, and the latter two would (probably) produce the same result, but those wouldn’t be the same as for the first two.

There are two reasons for this. First of all, the C and C++ isspace functions include a couple of often forgotten whitespace characters – the vertical tab ('\v', 0x0b) and the form feed ('\f', 0x0c). They don’t tend to see that much use nowadays, but are still defined as whitespace in both the C and C++ standards.

The second reason the results from isspace may differ from a hard-coded solution is that they are both dependent on what locale is in use. A changed locale will never indicate that any of the standard list of whitespace characters (" \t\r\n\v\f") is not a whitespace character, but may indicate that some further characters are also whitespace.

Since the functions already exist in the standard, it’s rather silly of us to write our own, so let’s just use isspace. Unless you muck about and change locales (and let’s not, if we can avoid it), both the C and C++ version behave the same way, so which you use is up to you.

Knowing how to identify whitespace characters, we only need to remove them. How do we do that? Well, that depends on whether we want to modify the string, or create a copy. In either case, let’s avoid the simplistic, completely hand-made solutions again:

// Working on std::string str

// Altering original
std::string::size_type p = 0;
while (p < str.size())
{
  // If character at p is space erase it, otherwise go to next
  if (isspace(str[p]))
    str.erase(p, 1);
  else
    ++p;
}

...

// Making a copy
std::string output;
for (std::string::size_type i = 0; i < str.size(); ++i)
{
  if (!isspace(str[i]))
    output += str[i];
}

Both these solutions work, but there are well established and standardised ways of doing these things using algorithms:

// Working on std::string str

// Altering original
str.erase(std::remove_if(str.begin(), str.end(), 
  &::isspace), str.end());

...

// Making a copy
std::string output;
std::remove_copy_if(str.begin(), str.end(), 
  std::back_inserter(output), &::isspace);

Simple!

No? Ok, let’s break it up. The functions in the C++ <algorithm> header generally work on three types of parameters: iterators, predicates and function objects (aka functors). In the code above, we’re not using any functors, so we’ll put them aside for the moment.

&::isspace – predicate. This is simply a pointer to a function that takes one parameter and returns a bool, in this case indicating whether a given character is whitespace or not, as discussed earlier.

str.begin(), str.end() – iterators, in this case indicating where to start and stop running the algorithm. We want to go through the whole string, so we start at the beginning, and end at the, well, end.

str.erase(std::remove_if(...), str.end()); – this is the erase-remove idiom. Because the remove_if function only takes iterators, it can’t actually remove anything. What it can do is re-shuffle, and put all the elements (or characters in the string, in this case) that match the predicate (is whitespace) at the end of the given range. It then returns an iterator that gives the first position of these predicate-fulfilling characters. This iterator is then given to the erase member function of the string, as the start of the characters to erase, and str.end() as the end.

std::back_inserter – iterator. This is a handy little helper that gives an output iterator for the given container (i.e. an iterator that can be used to insert elements in a containiner). (Unfortunately, Microsoft’s documentation still says the container given to it must be a std::vector, std::list or std::deque, which is not true. The only thing required is that the container has the member function push_back, which std::string does. Given how popular their development tools are, it’s surprising this hasn’t been amended.)

std::remove_copy_if – this is an amazimgly poorly named function, which ought to be called std::copy_if_not. What it does is: go through the range given (i.e. begin to end), call the predicate (i.e. isspace) with each element in the range, and if the predicate returns true, don’t copy it. It doesn’t remove anything from the input range (it can’t, as it only has iterators), and in fact doesn’t change anything at all on the range it’s given. I guess that conceptually, it removes an element for which the predicate is true from a list of elements to copy. Except, there is no such list. In short: horrible name, copies elements not fulfilling the predicate.

So, there we are. Two simple and useful functions to remove whitespace:

void remove_whitespace(std::string& str)
{
  str.erase(std::remove_if(str.begin(), str.end(), 
    &::isspace), str.end());
}

void remove_whitespace(const std::string& input, std::string& output)
{
  output.clear();
  std::remove_copy_if(input.begin(), input.end(), 
    std::back_inserter(output), &::isspace);
}

(Of course, if you really want to use std::isspace with std::locale, things start to get a bit… well, complicated. I might return to that at some later point.)

Nowadays, we know better, and don’t write new code to solve problems that have already been solved. Well, unless we need an XML parser – everybody (including myself, but that’s a post for another time) has written one of those – or at least a string tokeniser (aka splitter).

Other languages get tokenisers for free (C# – String.Split, Java – String.split, Python – string.split, and so on, and even C has strtok), but not C++. Which is why it’s something almost every C++ programmer writes, at some point or other.

Of course, you can use the rather nifty boost::tokenizer, if the place where you work is okay with using Boost (a surprising number of places aren’t, for various reasons), or find one of the numerous example implementations out there. Like this one, for instance:

void tokenise_string(const std::string& str, 
  const std::string& separator, 
  std::vector<std::string>& tokens, 
  bool empty /* = false */)
{
  const std::string::size_type strlength = str.length();
  const std::string::size_type seplength = separator.length();

  std::string::size_type prev = 0;
  std::string::size_type next = str.find(separator, prev);

  while (std::string::npos != next)
  {
    if (empty || prev != next)
      tokens.push_back(str.substr(prev, next - prev));
    prev = next + seplength;
    next = str.find(separator, prev);
  }
  if (empty || prev != strlength)
    tokens.push_back(str.substr(prev, strlength - prev));
}

There’s not that much to say about this. Pass in a string to split up into tokens, what separator to look for, and an output parameter which will hold the tokens when we’re done. What makes this implementation slightly different from some the others is that the separator is a std::string, and treated as such. Other implementations I’ve seen take a char (or even std::string::value_type) as a separator, or a string which is treated as a list of possible separators (like “.!?” to split a text into sentences).

I dislike the latter, as it’s ambiguos – is the separator used as a full string or as an array of characters? Rather, I’d prefer to make it explicit by overloading the function

void tokenise_string(const std::string& str, 
  const std::vector<std::string::value_type>& separators, 
  std::vector<std::string>& tokens, 
  bool empty /* = false */)
{
  const std::string::size_type strlength = str.length();
  const std::string::size_type seplength = 1;
  const std::string sep(separators.begin(), separators.end());

  std::string::size_type prev = 0;
  std::string::size_type next = str.find_first_of(sep, prev);

  while (std::string::npos != next)
  {
    if (empty || prev != next)
      tokens.push_back(str.substr(prev, next - prev));
    prev = next + seplength;
    next = str.find_first_of(sep, prev);
  }
  if (empty || prev != strlength)
    tokens.push_back(str.substr(prev, strlength - prev));
}

However, there is a problem here, in that std::string can be implicitly created from a native array of characters, and std::vector can’t:

std::vector<std::string> output;
std::string input = "What, me worry? Nah.";
char separators[] = {'.','?'};

// Will call std::string separator version, which we probably don't intend
tokenise_string(input, separators, output);

// Must set up a vector explicitly
std::vector<char> sep_array(&separators[0], &separators[2]);

// Will call std::vector separator version
tokenise_string(input, sep_array, output);

For now, that is. C++ 1x will have an initializer_list constructor which will make things interesting here.

By the way, the benefit of treating a separator string as one single separator is, of course, that it lets us parse telegrams:

std::vector<std::string> output;
std::string input = "NO TIME FOR WRENCHES STOP HAMMER TIME STOP";
std::vector<std::string> separators = "STOP";
tokenise_string(input, separators, output);
// Now output has two strings

I should probably mention, too, that the empty parameter lets us specify whether to include empty tokens in the output. In most cases, I don’t want to, but there are times it’s significant, if only to indicate whether the string started or ended with a separator.

Finally, here’s a function you see implemented and talked about a lot less often than its counterpart. If you want to split, presumably you’ll also want to merge, at some point. While it’s a very simple function, I’ve found it handy to have it available, so the merging is consistently done:

void merge_tokens(const std::vector<std::string> &tokens, 
  const std::string& separator, 
  std::string& output)
{
  if (!tokens.empty())
  {
    output = tokens.front();
    for (std::vector<std::string>::const_iterator i = 
      ++(tokens.begin()); 
      i != tokens.end(); ++i)
    {
      output += separator + *i;
    }
  }
}

Here we see the difference the empty flag makes in a call, by the way:

std::vector<std::string> split1, split2;
std::string input = "/usr/tmp";
std::string separator = "/";

tokenise_string(input, separator, split1);
tokenise_string(input, separator, split2, true);

std::string merged1, merged2;

merge_tokens(split1, separator, merged1);
merge_tokens(split2, separator, merged2);

assert(input != merged1);  // Initial / removed
assert(input == merged2);  

I think I first tried using a std::stringstream, but gave up on that as being too slow, and went with strtoul instead. I might also have played around with using a std::map lookup table, with all the headaches that brought in terms of storage and initialisation, and decided against it.

What I didn’t try was a straight, non-clever switch-based lookup table to find the integer value of a hexadecimal character digit:

inline unsigned char hex_digit_to_nybble(char ch)
{
  switch (ch)
  {
    case '0': return 0x0;
    case '1': return 0x1;
    case '2': return 0x2;
...
    case 'f': return 0xf;
    case 'F': return 0xf;
    default: throw std::invalid_argument();
  }
}

I did when writing that post, though, and found that this was twice as fast as using strtoul. That in itself isn’t surprising, as it’s a much simpler functional requirement to fulfill (the standard strtoul is a very capable parser).

As it happens, using this digit-to-nybble function is not only faster, but also simplifies the main conversion function calling it. For one thing, this function handles all sanity checking and validation of the characters given, and for another, it removes the need to copy the two characters into a hex string before doing the conversion.

Faster, neater, tidier code – brilliant!

However, I was troubled by one question: what character set does C++ use?
See, I was using literal characters, and we’ve all been warned about making assumptions about character sets. I even referred to that kind of dangerous assumption in the post I wrote, without, if I’m honest, thinking much about it.

Looking at the code again, I realised that I was already assuming things about the character set, since I use an array of characters when encoding integer values into hex strings:

static const std::string hex_char = "0123456789abcdef";
...
char high_nybble_char =  hex_char[[(data_char >> 4) & 0x0f];
char low_nybble_char =  hex_char[[data_char & 0x0f];

In other words, if this was safe, the switch was safe. If it wasn’t, neither was the switch, and while I already had a portable safe and working version for the decoding (using strtoul), I might have to come up with an alternative for the encoding.

Essentially, for this code to be guaranteed to be portable, standards-compliant, and safe, three potentially different character sets need to agree that the sixteen characters used to represent hexadecimal numbers are the same. Those three are:

1. The character set used to store the source code files,
2. The character set used by the compiler when creating executable code,
3. The character set used by the data sent into the decoding function, and accepted as output by the caller of the encoding function.

The problem is that a character is only a character when printed or written or displayed. It’s a visual shape, a representation. This A (probably) looks like a character to you, but it isn’t stored as one. It’s stored in some sort of binary representation, using some defined rule set and look-up that tells your computer (or mobile, iPad, text-to-speech system, etc) that it should be rendered in a manner that represents the concept of the letter “A” in one of the many Latin alphabets. Probably.

So the compiler needs to be able to read the source file, which is a stream of bytes (which, of course, may be of any number of lengths, although 8 bits is the most common), and interpret those into a representation where, for instance, the bytes {0×69, 0×66, 0×20, 0×28} are parsed as “if (” and not “%ö”.

Of course, this is what a compiler is required to do, and regardless of how the source code is stored (well, provided it’s in a form the compiler suports), the compiler will read it and convert it to “basic source character set”. This character set includes all hexadecimal digits, so we’re safe there. (It actually includes most of your basic printable 7-bit ASCII. There are a couple of good explanations of this on StackOverflow.)

Next, the compiler has to worry about the “basic execution character set”, which is what character set – at minimum – is used during execution. This is defined as the “basic source character set” plus a few extra characters, so we’re good there, too.

Finally, the data sent in to the function is safe, too, because the “basic source character set” covers the calling functions too, and in the case of data read from disk or otherwise externally sourced, it is the responsibility of the calling function to provide it in that form.

For more information on these character sets, see the C++ standard, or these two articles, which have been very helpful to me:
What character set does C++ use?
C++ Character Sets

So, anyway, it’s safe to use character literals to represent hexadecimal digits, which means that the bytes_to_hex function can remain unchanged, and the hex_to_bytes can be rewritten to use the much faster switch lookup:

inline unsigned char hex_digit_to_nybble(char ch)
{
  switch (ch)
  {
    case '0': return 0x0;
    case '1': return 0x1;
...
    case 'f': return 0xf;
    case 'F': return 0xf;
    default:
    {
      std::string e = "Invalid character in hex string: \'";
      e += ch;
      e += "'";
      throw std::invalid_argument(e);
    }
  }
}

void hex_to_bytes2(const std::string& str, 
  std::vector<unsigned char>& data)
{
  // Sanity check
  static_assert<8 == CHAR_BIT>::valid_expression();
  
  // Clear output
  data.clear();
  
  // No data? Then we're done
  if (str.empty())
    return;

  // Must be prepared that string can have odd number of 
  // nybbles, in which case the first is treated like the low 
  // nybble of the first byte
  size_t lengthOverflow = str.length() % 2;

  // This also affects the length of the data buffer we
  // allocate (need full  byte for nybble)
  const size_t length = lengthOverflow + str.length() / 2;
  data.resize(length);

  // Buffer for byte conversion
  static char buf[3];
  buf[2] = 0;
  // End of input
  char* pend = &buf[2];

  // Iterators for input and output
  size_t i = 0;
  size_t c = 0;

  // If the first nybble is a low, we'll do it separately
  if (1 == lengthOverflow)
  {
    data[i++] = hex_digit_to_nybble(str[c++]);
  }
  
  // For each output byte, we use two input characters for 
  // high and low nybble, respectively
  while (i < length)
  {
    data[i++] = (hex_digit_to_nybble(str[c++]) << 4) |
      hex_digit_to_nybble(str[c++]);
  }
}

So there we are. I spent an evening learning a bit more about C++, improving some old code, and writing this post saying “You can always improve”.

The simplest encoding is (in my opinion) the “hexadecimal text” encoding. It’s so simple, it doesn’t even have a fancy or clever name. You simply take each byte and type its value as a hexadecimal number. Working on the assumption that a byte is 8 bits, its value can be expressed in two characters – 0×00-0xff. Assuming that a character occupies one byte, we see that the size of the data will double by writing it as hexadeximal text, so it’s not very efficient space-wise. But it is simple to understand and implement, and quite useful, so I wrote a pair of encoding/decoding functions.

Let’s start with the encoding function, as that’s simplest. I’ll use std::string to store the resulting string here, with the above assumptions. (Those assumptions – 8-bit memory bytes, 8-bit characters – are quite reasonable, in that if you’re working on a platform where they’re not true, you probably know about it.)

#include <limits> // For char size

// Encode data buffer to string of hexadecimal values
void bytes_to_hex(const std::vector<unsigned char>& data, 
  std::string& str)
{
  // Just wrapping the more "raw" function
  bytes_to_hex(&data[0], data.size(), str);
}

void bytes_to_hex(const unsigned char* data, size_t length,
  std::string& str)
{
  // Sanity check
  static_assert<8 == CHAR_BIT>::valid_expression();
  
  // Clear output
  str.clear();
  
  // No data? Then we're done
  if (0 == length)
    return;

  // Output is twice the length of input length
  str.resize(length * 2, ' ');
  
  // Working with 4-bit nybbles, we can use the value as
  // index to character
  static const std::string hex_char = "0123456789abcdef";

  for (size_t i = 0; i < length; ++i)
  {
    // High nybble
    str[i<<1] = hex_char[(data[i] >> 4) & 0x0f];
    // Low nybble
    str[(i<<1) + 1] = hex_char[data[i] & 0x0f];
  }
}

As you see, it’s very simple. Given a buffer of bytes {7, 233, 57, 42, 198} the string “07e9392ac6″ is generated into the output parameter.

While there is a standard way of turning a number into a string – with std::stringstream – I elected to write the code to do it myself here, since it’s a very simple and safe algorithm.

First, I set up an array of the sixteen hexadecimal digits, and then simply use the high and low nybble as an index into this array to get the character corresponding to the value of the nybble. The high nybble has to be shifted down to get the correct range, and that’s all there is to it.

Since I had already resized the output string, I can write directly into the correct position, instead of appending (which would likely be significantly slower).

It’s tempting to write the decoding function as a straight reverse, but this stumbles on the character-to-value lookup. How do you get from a character to its corresponding nybble? The naive solution looks as follows:

std::string hex = "f3"; // For instance
...
char hi_nybble = hex[0];
char lo_nybble = hex[1];
unsigned char result = 0;

// First for high nybble, then low
// Numeric or alphabetic?
if (hi_nybble > '9')
  result |= (hi_nybble - 'a' + 0xa) << 4;
else
  result |= (hi_nybble - '0') << 4;
if (lo_nybble > '9')
  result |= (lo_nybble - 'a' + 0xa);
else
  result |= (lo_nybble - '0');
...

Ignoring for the moment that there’s no sanity checking of the input data, assuming only the characters [0-9,a-f] will be present, this function still fails the “good engineering” test by making assumptions about character ordering and values. It’s not safe to use, and may stop working when used with a different character set.

An alternative would be to make a proper lookup table, mapping characters to nybble values, with separate entries for upper and lower case characters (a-f), either by populating a std::map or a big switch:

inline unsigned char hex_digit_to_nybble(char ch)
{
  switch (ch)
  {
    case '0': return 0x0;
    case '1': return 0x1;
    case '2': return 0x2;
...
    case 'f': return 0xf;
    case 'F': return 0xf;
    default: throw std::invalid_argument();
  }
}

Then, after I have the nybbles, I could shift the high one up, and do a bitwise OR to join them. But frankly, while this works, it feels clunky. And besides, there are lots of standard ways to convert a string to a number; from the standard C library functions atoi and strtol, to the standard C++ std::stringstream (and even boost::lexical_cast which isn’t standard, but fairly popular). However, only two of those can handle numbers in bases other than decimal – strtol and std::stringstream – and of those, the latter is much more powerful, and therefore likely to be slower.

The strtol function expects a character string, so I’ll have to copy each pair of characters into a zero-terminated buffer, and use that as input to get a byte. That’s simple enough, but what do I do if there isn’t a pair of characters, but a single one?

In other words, if I have the hex string “3da” to convert, the function should treat it like “03da”, and produce {03, da} rather than {3d, a0}. Rather than making a copy of the string with an extra “0″ prepended, I’ll treat this as a special case.

Any other potential problems with using strtol? Well, yes, the matter of what characters count as valid input. I’ll use the unsigned version, strtoul, since I’m expecting an unsigned output, but even this is far too lenient in what it accepts: [whitespace][{+|–}] [0[{x|X}]][digits].

Since I’m not converting numbers, but encoding bytes, I can’t accept any whitespace, signs, or anything that isn’t a hexadecimal digit. Looking into this further, it’s less of a problem than it would first appear, as strtoul will let me know if there’s an un-parsed character at the end. In other words, “-3″ is fully parsed, but for “3-” it will only parse the first digit and then stop, so that’s a simple thing to check for. Furthermore, by telling it explicitly what base to use, it will disallow an initial “0x”.

Still, we need to make sure the initial character is valid hex, which means breaking out isxdigit to make sure we only accept hexadecimal digits. Now, this is a function that is both available from the standard C library, via the <cctype> header, and from the standard C++ library, via the <locale> header. The difference is that the std::isxdigit takes a std::locale, which I assume is just for completeness’ sake, as the C isxdigit is not affected by any changes to locale. At best, it’s just a through call only adding a level of indirection, and at worst, it adds more unnecessary computing, so I’ll just stick to the C version.

#include <cctype> // For isxdigit

void hex_to_bytes(const std::string& str, 
  std::vector<unsigned char>& data)
{
  // Sanity check
  static_assert<8 == CHAR_BIT>::valid_expression();
  
  // Clear output
  data.clear();
  
  // No data? Then we're done
  if (str.empty())
    return;

  // Must be prepared that string can have odd number of 
  // nybbles, in which case the first is treated like the low 
  // nybble of the first byte
  size_t lengthOverflow = str.length() % 2;

  // This also affects the length of the data buffer we
  // allocate (need full  byte for nybble)
  const size_t length = lengthOverflow + str.length() / 2;
  data.resize(length);

  // Buffer for byte conversion
  static char buf[3];
  buf[2] = 0;
  // End of input
  char* pend = &buf[2];

  // Iterators for input and output
  size_t i = 0;
  size_t c = 0;

  // If the first nybble is a low, we'll do it separately
  if (1 == lengthOverflow)
  {
    buf[0] = '0';
    buf[1] = str[c++];
    unsigned char x = static_cast<unsigned char>
      (strtoul(buf, &pend, 16));
    
    // Parsing should stop at terminating zero
    if (pend != &buf[2])
    {
      std::string e = "Invalid character in hex string: \'";
      e += *(pend);
      e += "'";
      throw std::invalid_argument(e);
    }
    data[i++] = x;
  }
  
  // For each output byte, we use two input characters for 
  // high and low nybble, respectively
  for (; i < length; ++i)
  {
    buf[0] = str[c++];
    // strtoul accepts initial whitespace or sign, we can't
    if (!isxdigit(buf[0]))
    {
      std::string e = "Invalid character in hex string: \'";
      e += buf[0];
      e += "'";
      throw std::invalid_argument(e);
    }

    buf[1] = str[c++];
    unsigned char x = static_cast<unsigned char>
      (strtoul(buf, &pend, 16));

    // Parsing should stop at terminating zero
    if (pend != &buf[2])
    {
      std::string e = "Invalid character in hex string: \'";
      e += *(pend);
      e += "'";
      throw std::invalid_argument(e);
    }

    data[i] = x;
  }
}

(As it happens, when writing this post I came across a very interesting blog entry by Tino Didriksen testing different methods of converting strings to integers in C++, using decimal strings, so all the methods I mention above are timed. Looking at those results there’s little to recommend std::stringstream in terms of speed, which was a nice validation of code I wrote the first version of years ago.

I also used the benchmark code from Tino as a to compare the usage of strtol to the hex_digit_to_nybble outlined above, and found that the latter was almost twice as fast. I’m currently pondering what drawbacks there might be in using what was quickly concieved as a rethorical strawman while writing this article.)

As so often happens, just having bounds available made me think of what variants of it could be useful. For instance, it would be handy to have it work for floating point or non-POD types, which isn’t possible as it is written. Since the bounds class uses ‘non-type template parameters‘ for its limits, only integer types and enums are accepted.[1]

Even disregarding this restriction, I found that I had use for a dynamic range class, as opposed to the static bounds which has its boundaries set at compile time. Just a simple one, and like std::pair only having two values, but with both of the same type, and with them guaranteed to be ordered.

The last part there would make it a bit more complex than the simple std::pair struct, as I’d need to validate the values given in order to ensure that the minimum was lower than or equal to the maximum, but still, a simple enough little class.

template <typename T,               // data type
  typename L = less_than_comparison::closed<T>, // lower comparer
  typename U = less_than_comparison::closed<T> >// upper comparer
class range
{ 
  // Member data
  T minimum_, maximum_;
protected:
  // Validation function
  virtual void throw_if_invalid(const T& minimum, 
	const T& maximum)
  {
    if (maximum < minimum)
      throw std::invalid_argument("Minimum > maximum");
  }
public:
  // Type name for templated type
  typedef typename T type;

  // Default constructor
  range()
    : minimum_(T()),maximum_(T()) 
  {}
  // Assignment constructor
  range(T min, T max)
    : minimum_(min),maximum_(max)
  {
    throw_if_invalid(minimum_, maximum_);
  }

  // Get minimum value
  T get_minimum() const
  {
    return minimum_;
  }
  // Get maximum value
  T get_maximum() const
  {
    return maximum_;
  }
  // Set mimimum value 
  void set_minimum(T min)
  {
    throw_if_invalid(min, maximum_);
    minimum_ = min;
  }
  // Set maximum value 
  void set_maximum(T max)
  {
    throw_if_invalid(minimum_, max);
    maximum_ = max;
  }

  // Equality comparison operator
  bool operator==(const range& other) const
  {
    return (minimum_ == other.minimum_) && 
      (maximum_ == other.maximum_);
  }
  // Inquality comparison operator
  bool operator!=(const range& other) const
  {
    return !operator==(other);
  }

  // Get size of range
  int width() const 
  {
    return maximum_ - minimum_;
  }

  // Check if value is in range
  bool in_range(T val) const
  {
    return L::less(minimum_, val) && U::less(val, maximum_);
  }
  // Check if other range is subset of this
  bool in_range(const range& other) const
  {
    return L::less(minimum_, other.minimum_) && 
      U::less(other.maximum_, maximum_);
  }

  // Check if other range intersects with this
  bool intersects(const range& other) const
  {
    return 
	  (in_range(other.minimum_) || other.in_range(minimum_)) &&
      (in_range(other.maximum_) || other.in_range(maximum_));
  }
  // Create union of this and other range
  range make_union(const range& other) const
  {
    if (!intersects(other))
      throw std::invalid_argument("No union of ranges");
  
    return range(std::min(minimum_, other.minimum_), 
      std::max(maximum_, other.maximum_));
  }
  // Create intersection of this and other range
  range make_intersection(const range& other) const
  {
    if (!intersects(other))
      throw std::invalid_argument("No intersection of ranges");

    return range(std::max(minimum_, other.minimum_), 
      std::min(maximum_, other.maximum_));
  }
};
// Create intersection of two ranges
template <typename T>
range<T> operator&(const range<T>& lhs, 
  const range<T>& rhs)
{
  return lhs.make_intersection(rhs);
}

// Create union of two ranges
template <typename T>
range<T> operator|(const range<T>& lhs, 
  const range<T>& rhs)
{
  return lhs.make_union(rhs);
}

This uses the same policy-based design with upper and lower comparators as the bounds class, so that you can have an open, closed, or half-open (either directions) range.

Note that despite the inclusion of union and intersection functions and operators, this is not a class intended for interval arithmetic. If you have such needs, you’re much better off with the boost::interval class.

There is one virtual function in the range class: the validation function. The reason for this is that I can simply combine this with the bounds class into a bounded range, and only need to update the validation to take the bounds into consideration to have a fully functioning class. Well, that, and write suitable constructors, and provide another bounds-checking function.

template <typename T, T lower_, T upper_, 
  typename L = less_than_comparison::closed<T>, 
  typename U = less_than_comparison::closed<T> >
class bounded_range : public range<T, L, U>, 
  public bounds<T, lower_, upper_, L, U>
{
protected:
  /*  Overridden validation function to check bounds as well 
      as validity.
    Throws if minimum > maximum, or if out of bounds
    \param minimum lower value of range
    \param maximum upper value of range
  */
  virtual void throw_if_invalid(const T& mini, const T& maxi)
  {
    range<T, L, U>::throw_if_invalid(mini, maxi);
    if (!in_bounds(mini))
      throw std::invalid_argument("Minimum out of bounds");
    if (!in_bounds(maxi))
      throw std::invalid_argument("Maximum out of bounds");
  }
public:
  // Type name for templated type
  typedef typename T type;
  // Default constructor
  bounded_range()
    : range(lower_bound(), upper_bound())
  {}
  // Assignment constructor
  bounded_range(const T&  min, const T&  max)
    : range(min, max)
  {
    throw_if_invalid(min, max);
  }
  // Conversion constructor
  bounded_range(const range<type>& other)
    : range(other)
  {
    throw_if_invalid(other.get_minimum(), other.get_maximum());
  }
  // Check if value is within bounds using base class
  using bounds<T, lower_, upper_>::in_bounds;
  // Check if range is within bounds
  static bool in_bounds(const range<T>& other)
  {
    return in_bounds(other.get_minimum()) && 
	  in_bounds(other.get_maximum());
  }
};

[1] The C++ language also permits address types (pointer or reference) as non-type parameters, provided they’re known at compile time, but for that loophole to provide a way to implement static bounds checking with float or, say, std::point types, would, if at all possible, require a mastery of template metaprogramming magic that is far beyond my meagre abilities.Back

Most of the things in the bag, though, has been unorganised little snippets of code, random functions and classes that have been half-formed, often neither tidy or documented enough to be presented publicly. Some has come from experiments and trials, some from various hobby projects, and some from times my work contract has given me some rights to the software I write.

The last part there is a sensitive area. In most employment contracts, the employer retains the full right to the code you write as part of your job. If you are an independent contractor, the rights issue can be more flexibly negotiated, and there’s sometimes a distinction made between business-related code, and supporting code.

In most cases, the code I’ve written in my day job is code I have no rights to. But because I’m a geek, I’ve sometimes thought “Oooh, that’s interesting”, and sat down to try out and experiment with some concept in my spare time. In those cases, the inspiration might have come from my day job, but the code has been written in my spare time, from scratch. It may be because I’ve wanted to learn more, or because it’s been an interesting problem to solve, or simply because the day job has only required a partial, specialised solution, so I’ve wanted to do a full solution for my own intellectual satisfaction.

It’s often been the case, too, that I’ve found a use for these little snippets later, for a different employer, and in those cases I’ve taken my old code and tweaked or rewritten it to fit into the style and needs of the current workplace.

So I finally decided to tidy things up. I’ll be re-visiting my old code folders, and extract what’s useful or interesting, and tidy it up, and package it. I will:

• stick to a single style of coding (per language) and rewrite the code to have a consistent look
• organise the code into namespaces and classes
• comment the code thoroughly
• use Doxygen or similar to extract the comments to useful documentation
• write a simple test app to exercise the code for each module
• for each module, create packages with:
  • only source code
  • source code, documentation, and example/test app
• release under a BSD license

Why? Well, in a large part to make it easier for me to find, and introduce at the places I work. And I also think other people might find these bits and bobs useful.

I wonder, though, whether it would also be useful to make the code available on SourceForge and/or CodeProject?