A Common Lisp Web Development Primer, Part 2

Tagged as LISP, Programming

Written on 2010-11-16 03:19:24

Disclaimer: This blog will not teach you Common Lisp. That is a (mostly) solved problem. See Peter Seibel's Practical Common Lisp and Peter Norvig's PAIP. Other oft-recommended texts include Keene's OOP in CLOS, PG's On Lisp and Kiczales et al's The Art of the Metaobject Protocol. This blog will also not teach you good style, I'm too young for that. It hopefully demonstrates non-atrocious style though. I'm learning web development as I go, so don't count on expert understanding there either.
Disclaimer Pt. 2: For the foreseeable future all these projects will be weblocks-based. If that's not your cup of tea you can check out the RESTAS docs and examples, Felideon's blog on UCW, Adam Petersen's slightly bitrotted sans framework article or "defect" to Clojure and look at all the Compojure stuff and maybe Sandbar.

Introduction

It's taken far longer than I hoped to get this second article off the ground. For those of you who missed Part 1, look here and if you'd rather see code than this article's commentary, the code is available on github. It's worth noting that Part 1 was originally written with clbuild in mind but has since been updated with instructions for quicklisp also. Part 2 details the construction of Clockwork, a simple clone of the now defunct yourli.st, an email reminder service. Clockwork allows you to schedule a reminder and brief note which is sent to you by email or text message at the predetermined time. Right now international numbers aren't supported but I'd be happy to see patches.

Future Plans

There's still a good amount of stuff in the TODO and I have further projects in mind after this. Part of the reason this article was so long in coming is that I've been helping Leslie by testing out his new form-widget library. The other reasons are that I have school and (until recently) was working plus I'm learning web development as I go. The next article will go into polishing this application and will likely be much more Javascript and CSS than Lisp. I'm working on getting a postmodern backend written, tested and merged into Weblocks right now. Once that's done I plan to continue this series by developing a RESTful blog that can import entries from Wordpress and maybe crosspost to livejournal as that should prove more interesting...but like Linus' said, "Talk is cheap, show me the code".

Resources, Libraries and Project Skeleton

Weblocks docs are not in an ideal state and hopefully this blog series will help that some. Four things worth being aware of for a beginner are the Google group, the TINAA-generated docs, the User Manual and User Guide. As usual, googling specific concepts will lead you to blog entries and mailing list posts that may or may not be out of date.

We'll begin by using weblocks helper to create a project skeleton by evaluating (wop:make-app 'clockwork "/home/redline/projects/"). If you've been following along since Part 1, you'll also want to push that path onto the ASDF central registry so you can use quicklisp to load the clockwork system in your server init file (~/webapps/init.lisp). (push "/home/redline/projects/clockwork/" asdf:*central-registry*) will do the trick. Then add a (ql:quickload '(clockwork)) line at the bottom of the file followed by (clockwork:start-clockwork :port 4242). At this point, you should be able to reboot the server and run screen -dRR or similar to get a screen session with emacs and an sbcl instance with clockwork and swank running. They'll be in different windows which you can switch to with C-a (control-a) and the window number. Numbers start at 0. Enough of that, this isn't a GNU Screen tutorial. Go to the emacs instance and run M-x slime-connect making sure to change the port to that specified in the init file. At this point, you're connected to SLIME and can evaluate (in-package :clockwork) and finally get hacking! You should also be able to reach clockwork in the browser at localhost:4242 but there's not much there yet...

To begin with, you'll need some libraries to send emails and schedule reminders to be sent in the future. SBCL provides a Timers facility which we could use for this but it's usually worth doing a little extra work to write portable Common Lisp. To this end we'll use the trivial-timers library as a wrapper and cl-smtp for emails. We'd also like to handle timezones and time arithmetic properly so we'll use the local-time library for that. We'll also be doing some minor string handling which split-sequence winds up being an easy solution for so grab that too. Add those to the :depends-on clause in clockwork.asd in the project directory and then run (ql:quickload 'clockwork) at the REPL. Quicklisp will download and load the new libraries for you. It's that easy. Finally, add :local-time to the :use clause of the defpackage in clockwork.lisp and import the split-sequence symbol from the split-sequence package.

Data and Weblocks Stores

By default Weblocks defines a cl-prevalence backend ("store") which persists data to the "data/" directory in the clockwork project folder. The store itself is defined in conf/stores.lisp and that's where you would go to define additional stores if you wanted them. Weblocks has a special variable, *default-store*, and DEFSTORE sets that variable after defining a store so the last store defined in stores.lisp will act as the default. Weblocks also supports elephant and clsql but for now, we'll focus on other aspects of the framework and delve more into the Store API in a later article. If you're curious now though the Store API is clearly defined and documented.

The only data we really care about is the reminders our users will generate. For our purposes, a reminder consists of some number of recipients, a title, a summary of the event it's reminding you of, the time of the event and how far before the event you'd like to be reminded. A class definition falls out of this rather naturally and we'll add an id slot so prevalence will know how to store it. Create a src/reminder.lisp file, insert the following and add the file to the :components clause of clockwork.asd.

(in-package :clockwork)

(defclass reminder ()
  ((id :reader reminder-id) ;; all classes to be persisted with cl-prevalence need an id slot
   (emails :reader reminder-emails
           :initarg :emails
           :type list)
   (title :reader reminder-title
          :initarg :title
          :type string)
   (timestamp :reader reminder-timestamp
              :initarg :timestamp
              :type timestamp)
   (summary :reader reminder-summary
            :initarg :summary
            :type string)
   (at :reader reminder-at
       :initarg :at
       :type timestamp)))

Now that we have a rough idea of what data we care about, lets look at how to send messages.

Emails and Text Messaging

Text messaging is actually quite simple to support thanks to the SMS gateways run by the major carriers. SMS gateways allow us to send an email to an address which represents a phone number. This is then converted to a text message and forwarded on to the recipient's cell phone free of charge. The downside to this is that it's carrier specific so you have to know the cell carrier of the recipient. It would be nicer to just take a number and figure out what carrier services it but Local Number Portability, among other things, makes this tricky. Whitepages.com has an API for looking this up but their information was out of date for my cell phone and they had a 200 request per API key per day limit. Twilio and Data24-7 offer for-pay APIs but for this example app I'll be staying free and cheap. I'll be coldly forcing my users to select their carrier from a dropdown if they want SMS support.

Since we don't know whether our users really care about their privacy or what kind of data they'll be putting in these reminders, we'll do the responsible thing and send the emails via Encrypted SMTP. I'll be using a gmail account I registered for the service since Google provides free, encrypted SMTP on all their accounts. Let's write a quick helper macro for using it. Create a src/messaging.lisp file, insert the following and add it to the :components clause of clockwork.asd.

(in-package :clockwork)

(defparameter *mail-server* "smtp.gmail.com")

(defmacro with-encrypted-smtp ((&key to subject style
                               (from "cl.ockwork.webdev@gmail.com"))
                               &body body)
  `(cl-smtp:send-email ,*mail-server* ,from ,to ,subject
                       (if (eql ,style :html)
                           (with-html ,@body) ;; TODO: make a nicer render style
                           ,@body)
                       ;; it's worth noting send-email takes a :cc argument
                       :ssl :tls
                       :authentication '(,*smtp-user* ,*smtp-pass*)
                       ,@(when (eql style :html)
                               '(:extra-headers
                                 '(("Content-type"
                                    "text/html; charset=\"iso-8859-1\""))))))

Note that we haven't defined *smtp-user* or *smtp-pass* yet. There are two questions you should be asking. Why a macro and what is it doing? The why is debatable in this case. I wanted the syntax to jump out at me and read a certain way when I use the encrypted SMTP. That's all. The what is fairly straightforward. The macro is syntactically similar to with-open-file and others. It takes keyword arguments for the recipient, sender (with a default value), subject and style of the message along with a message as the body and then sends an email via encrypted SMTP (and cl-smtp's send-email function) with the credentials provided. If the style is :html, it goes to the trouble of specifying a few additional headers.

Since we haven't defined the user and pass, let's do that now. Create a conf/config.lisp file, insert the following and add it to your clockwork.asd.

(in-package :clockwork)

(defparameter *smtp-user* "yourusername@gmail.com")
(defparameter *smtp-pass* "yourpassword")

Obviously, you don't want this puppy in source control. Consequently, I committed it before I filled in the user and pass values then ran git update-index --assume-unchanged conf/config.lisp which tells git to ignore all future changes to the file. Be forewarned, that command might be reversible but I don't know how. Go ahead and add in your username and password, reload the system at the REPL with (ql:quickload 'clockwork) and test it out. You should be able to send yourself an email. Now let's add some helpers for SMS. Return to the src/messaging.lisp file and we'll add a variable defining a mapping of Carriers to SMS Gateway servers and a function for determining if an email address belongs to one of the listed SMS gateways. Add the following code to the bottom of the file.

(defparameter *sms-gateways*
  ;; list is derived from http://en.wikipedia.org/wiki/List_of_SMS_gateways
  '(("AT&T/Cingular" . "txt.att.net")
    ("Alltel" . "text.wireless.alltel.com")
    ("Boost Mobile" . "myboostmobile.com")
    ("Cincinatti Wireless" . "gocbw.com")
    ("MetroPCS" . "mymetropcs.com")
    ("Sprint/PCS" . "messaging.sprintpcs.com")
    ("Sprint/Nextel" ."page.nextel.com")
    ("T-Mobile" . "tmomail.net")
    ("US Cellular" . "email.uscc.net")
    ("Verizon" . "vtext.com")
    ("Virgin Mobile" . "vmobl.com")))

(defun sms-mail-p (email)
  (let ((domain (second (split-sequence #\@ email))))
    (member domain *sms-gateways* :key #'cdr :test #'equal)))

A Few Reminder Methods

We still need methods to send a reminder, delete it when we're through with it and schedule it to be sent at a later time. Let's create those now. Since users aren't required to register to send a reminder, we'll put off letting them delete reminders from the system for now. When they submit the reminder form, a reminder will be instantiated, persisted and scheduled for later sending and deletion. We'd like to offer the user the ability to send reminders by email, text message or both so we'll assume that a list of emails are stored in the emails slot of the reminder and loop through each one, sending it with the macro we defined earlier. Then we'll use Weblocks Store API to delete the object from the datastore. Add the following code to the end of src/reminder.lisp.

(defgeneric send-and-delete (reminder)
  (:documentation "Send the user their reminder as requested and then remove it from the datastore."))

(defmethod send-and-delete ((reminder reminder))
  (loop for email in (reminder-emails reminder) do
    (with-encrypted-smtp (:to email :subject (reminder-title reminder)
                              :style (if (sms-mail-p email)
                                         :plain
                                         :html))
      (reminder-summary reminder)))
  (delete-persistent-object-by-id *default-store* 'reminder (reminder-id reminder)))

Finally, we'd like to schedule the reminder to be sent at a later date. Assuming that the timezone differences are handled beforehand and that the reminder's at slot contains a timestamp for when the message should be sent according to the server's timezone, the local-time and trivial-timer libraries make defining a schedule method pretty easy. We'll just use the local-time library and a let to compute the seconds from the present until the time to send the reminder, make a timer whose function calls send-and-delete on the reminder and schedule it with a delay of the number of seconds computed. The only tricky bit is to pass :thread t to make-timer so that each timer is triggered in a new thread. If this isn't done, the timer will try to interrupt an arbitrary thread to run it's function which, put plainly, is unreliable. Another alternative would be to have a dedicated thread for sending reminders and pass that as the argument to :thread but we'll take the easy way out this time. Add the following code to the end of reminder.lisp. Here are links to my versions of the files: messaging, reminder.

(defgeneric schedule (reminder)
  (:documentation "Schedule the reminder to be sent at the time the user requested."))

(defmethod schedule ((reminder reminder))
  (let ((secs-until-reminder (round (timestamp-difference (reminder-at reminder) (now)))))
    (trivial-timers:schedule-timer
     (trivial-timers:make-timer (lambda ()
                                  (send-and-delete reminder)) :thread t)
     secs-until-reminder)))

Next Time...

This article got long. It may be too high-level for some, too low-level for others and too wordy for everybody. In addition, this isn't the most thrilling software ever constructed. Next time we'll get into more Weblocks specifics and work on the frontend to get the form and a jQuery calendar up and going. Please let me know if there are questions I can answer, things you'd like covered in more depth or other thoughts on how to improve this series. Thanks for reading.

• ← Previous
• Next →