A Common Lisp Web Development Primer, Part 3
Tagged as LISP, Programming
Written on 2010-11-22 04:24:03
Disclaimer: What? You haven't already read the first two parts? Feel free to go ahead and do that. The same disclaimers apply. (require 'cl-wdp-part1 'cl-wdp-part2)Today's Topics
The main topics we'll be covering are Weblocks widgets, forms and views along with a brief example of creating a presentation to use jQueryUI's Datepicker. There will also be a brief aside on what to do after an emergency reboot or power outage relating to cl-prevalence and trivial-timers. By the end the clockwork site will be fully functional.
Form-widgets, Widgets and Views
The User Guide has pretty nice expository summaries of Widgets and Views. The bottom line is that Widgets are what Weblocks pages are composed of and views are different ways to render those widgets. A macro called defwidget is used to construct widgets which are just new classes which inherit from the widget-metaclass. Don't worry if you don't know what that means. Essentially, you define widgets for the data you care about working with and then you define views to render that data, whether as a form for editing, a table for comparing multiple items or some other representation.
Leslie has been working on some new form-widget code intended to remove some of the sharp edges of the current system. As this site is basically just a form it made sense for me to try to use this new code (and Leslie wanted me to help him bang on it). Consequently, one thing you'll need to do is add (load "/path/to/weblocks/contrib/lpolzer/form-widget.lisp") to your init.lisp file that runs on startup. It should be placed after weblocks is loaded and before the loading of clockwork.
Last time we defined a reminder class with slots for the data we really care about: a list of emails, a title and summary and timestamps for when to send that message and when the event itself occurs. However, it won't do to ask our users to input timestamps or to tell us the "email" for their cell phone's SMS gateway. To that end, we'll define a view to gather the information we really need to construct the timestamps and other parts of the reminder. We need to know whether they want to be notified by email, text message or both. We'll need the corresponding contact info, including their cell carrier if they want to be notified by text message. We'll also need to note the event date, timezone and time as well as how long before the event they'd like to be reminded and a message and title. I've also thrown in a "honeypot" field. In theory, spam bots will indiscriminately fill it so we won't get any bogus submissions because the form won't validate. Maybe later we'll replace this with reCaptchas.
So let's get to it and define a view for our form in a new file src/forms.lisp. Insert the following code:
(in-package :clockwork)
(defview reminder-form-view (:type form :caption "Schedule an Event Reminder..."
:buttons '((:submit . "Submit")) :persistp nil)
(send-as :present-as (dropdown :choices '(("An email and a text." . :both)
("Just an e-mail." . :email)
("Just a text." . :text))
:welcome-name "How to send it")
:requiredp t)
(email :satisfies 'valid-email)
(cell-number :satisfies 'valid-cell-number)
(cell-carrier :present-as (dropdown :choices *sms-gateways*))
(event-date :present-as (calendar) :requiredp t)
(event-hour :present-as (dropdown :choices *hour-choices*)
:requiredp t)
(event-minute :present-as (dropdown :choices '(("00" . 0)
("15" . 15)
("30" . 30)
("45" . 45)))
:requiredp t)
(timezone :present-as (dropdown :choices *timezones*)
:requiredp t)
(remind-me :present-as (dropdown :choices '(("At the event" . 0)
("5 minutes before" . 300)
("10 minutes before" . 600)
("15 minutes before" . 900)
("30 minutes before" . 1800)
("45 minutes before" . 2700)
("1 hour before" . 3600)
("2 hours before" . 7200)
("1 day before" . 86400)
("2 days before" . 172800)
("1 week before" . 604800)
("2 weeks before" . 1209600)))
:requiredp t)
(subject :requiredp t)
(summary :present-as (textarea :rows 5))
(honeypot :label "Leave this blank" :satisfies #'null))
(defparameter *timezones*
'(("UTC-12:00 (Eniwetok, Kwajalein)" . -43200)
("UTC-11:00 (Midway Island, Samoa)" . -39600)
("UTC-10:00 (Hawaii)" . -36000)
("UTC-09:00 (Alaska)" . -32400)
("UTC-08:00 (Pacific Time)" . -28800)
("UTC-07:00 (Mountain Time)" . -25200)
("UTC-06:00 (Central Time)" . -21600)
("UTC-05:00 (Eastern Time)" . -18000)
("UTC-04:00 (Atlantic Time, Caracas)" . -14400)
("UTC-03:30 (Newfoundland)" . -12600)
("UTC-03:00 (Brazil, Buenos Aires, Georgetown)" . -10800)
("UTC-02:00 (Mid-Atlantic)" . -7200)
("UTC-01:00 (Azores, Cape Verde Islands)" . -3600)
("UTC+00:00 (Lisbon, London, Casablanca)" . 0)
("UTC+01:00 (Berlin, Brussels, Copenhagen, Madrid, Paris)" . 3600)
("UTC+02:00 (Kaliningrad, South Africa)" . 7200)
("UTC+03:00 (Baghdad, Moscow, Riyadh, St. Petersburg)" . 10800)
("UTC+03:30 (Tehran)" . 12600)
("UTC+04:00 (Abu Dhabi, Baku, Muscat, Tbilisi)" . 14400)
("UTC+04:30 (Kabul)" . 16200)
("UTC+05:00 (Ekaterinburg, Islamabad, Karachi, Tashkent)" . 18000)
("UTC+05:30 (Bombay, Calcutta, Madras, New Delhi)" . 19800)
("UTC+05:45 (Kathmandu)" . 20700)
("UTC+06:00 (Almaty, Colombo, Dhaka)" . 21600)
("UTC+07:00 (Bangkok, Hanoi, Jakarta)" . 25200)
("UTC+08:00 (Beijing, Hong Kong, Perth, Singapore)" . 28800)
("UTC+09:00 (Osaka, Seoul, Sapporo, Tokyo, Yakutsk)" . 32400)
("UTC+09:30 (Adelaide, Darwin)" . 34200)
("UTC+10:00 (Eastern Australia, Guam, Vladivostok)" . 36000)
("UTC+11:00 (Magadan, New Caledonia, Solomon Islands)" . 39600)
("UTC+12:00 (Auckland, Fiji, Kamchatka, Wellington)". 43200)))
(defparameter *hour-choices*
(loop for i from 0 to 23
collecting `(,(format nil "~d" i) . ,i)))
So here we're defining a view called reminder-form-view and passing in a list of arguments *about* the view as well as a list of fields in the view. In the list of arguments about the view we note that it's a form and we don't want to persist the form contents directly. We use a variety of keywords in the list of form fields to get the behavior we want including :present-as, :requiredp, :satisfies and :label. Present-as allows us to make something a dropdown or any other defined presentation. Note that some presentations do take arguments. Dropdown in particular takes a list of dotted pairs representing the Dropdown choice and it's corresponding value. Requiredp does what you'd expect and marks a form field as required. Satisfies takes a lambda or the name of a function which will validate the field's data. By default, the view will "humanize" the field names and use those humanized names as labels. If you want a different label for some reason, you can achieve that with the :label keyword.
Now we have a form that takes all the data we need to construct a reminder but we still need to validate the emails and cell phone numbers. Additionally, we'll need to write helper functions to construct the email list and timestamps that the reminder's emails and timestamp slots will be set to. Consequently, add this code to the bottom of the file:
(defun valid-email (user-input)
"Ensure that there is an @ and a . and input not containing @s before and after each."
(or (cl-ppcre:scan "^[^@]+@[^@]+\\.[^@]+$" user-input)
(values nil "Your email must have an @, a . and text before and after both.")))
(defun valid-cell-number (user-input)
"Ensure that only numbers are given and there are at least 10."
(or (cl-ppcre:scan "^[0-9]{10,}$" user-input)
(values nil "Your number must have only numbers and at least 10 of them.")))
(defun get-emails (form-data)
(with-form-values (send-as email cell-number cell-carrier) form-data
(let ((sms-mail (concatenate 'string cell-number "@" cell-carrier)))
;; this was an ecase with keywords but weblocks converts
;; the keywords to strings somewhere in form submission
(cond ((string= send-as "BOTH") (list email sms-mail))
((string= send-as "EMAIL") (list email))
((string= send-as "TEXT") (list sms-mail))))))
(defun get-timestamps (form-data)
(with-form-values (event-date event-hour event-minute
remind-me timezone) form-data
(let* ((hour (parse-integer event-hour))
(minute (parse-integer event-minute))
(reminder-time-period (parse-integer remind-me))
(timezone (parse-integer timezone))
(datestring (split-sequence #\- event-date))
(day (parse-integer (first datestring)))
(month (parse-integer (second datestring)))
(year (parse-integer (third datestring)))
(event-time (encode-timestamp 0 0 minute hour day month year :offset timezone)))
(list event-time
(timestamp- event-time reminder-time-period :sec)))))
The validation functions are ORs with the function testing the input as the first clause and a VALUES form returning nil (a failed submission) and an appropriate error message as the second. The helper functions use the with-form-values macro to grab the relevant fields of the form and construct the resulting slot. Get-timestamps is rather nasty but we're essentially just grabbing all those fields pertaining to the time, parsing the integers from them and passing those on to the appropriate timestamp functions in the local-time library.
A Calendar Presentation
It would certainly be better to have a nice calendar than have users enter dates as strings and then try to validate them and God forbid we roll our own given the number of Javascript calendars already out there. Since it's fairly well established I opted for the jQueryUI Datepicker. Previously to use Javascript libraries you needed to download them and place them in your Weblocks project's pub/script folder but thanks to a quick patch by Leslie Polzer remote dependencies are now also supported. In case you didn't read the previous article, when you first start a project with
(wop:make-app 'name "/path/to/app") weblocks generates a defwebapp form and a basic package for that app along with setting up a store and some basic resources. To include the jQuery code on our page we'll modify our defwebapp form in clockwork.lisp like so:
(defwebapp clockwork
:prefix "/"
:description "Fire-and-Forget Event Reminders"
:init-user-session 'clockwork::init-user-session
:autostart nil ;; have to start the app manually
:ignore-default-dependencies nil ;; accept the defaults
:hostnames '("clockwork.redlinernotes.com")
:dependencies '((:stylesheet "http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.5/themes/ui-darkness/jquery-ui.css")
(:script "http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js")
(:script "http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.5/jquery-ui.min.js")
(:javascript-code "var $jquery = jQuery.noConflict();"))
:debug t)
First let's note that this will include these dependencies in the "HEAD" of every page. If you want to have per-widget dependencies Weblocks does support that by (if I'm not mistaken) defining a widget-public-dependencies method specialized on that widget. Since we only have one page in this app anyway we'll just list them here. We've done that by adding entries to the dependencies list that use Google's CDN to supply the minified jQuery and jQueryUI libraries along with a stylesheet for the jQueryUI stuff. We added the :hostnames argument which specifies that requests to any host besides those listed are to be ignored. This is particularly helpful if you're running multiple webapps off of one server but want them to share a Lisp image and port rather than fire up separate Hunchentoot servers for each one. Additionally, we're inlining the js code that calls
jQuery.noConflict() in the header because Weblocks uses prototype and scriptaculous out of the box and jQuery will happily steal the $ global variable from Prototype which causes all sorts of havoc. While there is interest in removing the prototype and scriptaculous dependencies it hasn't happened yet. It would be greatly appreciated if any developer felt like taking a little time to tackle this.So now that we've included the JS code, let's write a presentation. The presentation lets us add the code
:present-as (calendar) to a slot in our View and have it render as a calendar. This presentation will be a little different as we're using Leslie's new form-widget code. A more traditional coverage of presentations can be found in this blog post. Create a new file in the src directory called calendar.lisp and insert the following code:
(in-package :clockwork)
;; calendar presentation
(defclass calendar-presentation (input-presentation)
())
;; calendar form-widget code
(define-widget calendar-field-widget (field-widget)
()
(:default-initargs :parser (lambda (raw-value)
(values t raw-value))))
(defmethod field-presentation->field-widget-class ((presentation calendar-presentation))
'calendar-field-widget)
(defmethod render-field-contents ((form form-widget) (field calendar-field-widget))
(with-html
(:input :type "hidden" :name (name-of field) :value (datestring))
(:div :id "datepicker"
(send-script '($jquery (lambda ()
(ps:chain ($jquery "#datepicker")
(datepicker (ps:create date-format "dd-mm-yy"
min-date 0
on-select (lambda (date inst)
(ps:chain ($jquery "[name=event-date]")
(val date))))))))))))
(defun datestring ()
(subseq (format-timestring nil (now) :format '((:day 2) "-" (:month 2) "-" :year)) 0 10))
So what's going on here? First we define a calendar-presentation class and a calendar-field-widget class along with a simple method to map the presentation onto the widget-class. Then we write the bulk of the code, a render-field-contents method which generates the HTML and Javascript. We'll use a hidden input field with a name equal to the field in the view that gets initialized to today's date. That will be followed by a div containing the Javascript code for the datepicker written with Parenscript (indentation suggestions welcome) which sets the hidden input field whenever a date is selected.
Putting it all together...
Now that we have all the pieces we need we can hook them together. You'll note that a src/init-session.lisp file already exists and contains a defun for init-user-session. This function sets up the widget tree and creates a new session when users visit the site. Remove the old definition and insert the following into the file:
(defun init-user-session (root)
(setf (widget-children root)
(make-reminder-form)))
(defun make-reminder-form ()
(let ((reminder-form (make-instance 'form-widget :on-success 'submit-reminder-form)))
(form-widget-initialize-from-view reminder-form 'reminder-form-view)
reminder-form))
We're just defining a separate function
(make-reminder-form) to create our form instance here rather than defining it inline in the init-user-session code. Make-reminder-form itself creates an instance of the form-widget class which runs a function called submit-reminder-form when the form is successfully submitted (i.e. passes validation, etc). Note that we have not yet defined submit-reminder-form. Because the form is really based on a view and not a widget or class we want to persist we'll use form-widget-initialize-from-view in conjunction with the reminder-form-view we defined earlier. Note that you may need to restart the webapp after redefining the init-user-session function. Run (restart-webapp 'clockwork) and check the homepage. You should now have a nice form complete with jQuery Datepicker. But of course, nothing useful happens on submission. Time to fix that by going ahead and defining submit-reminder-form. Open src/init-session.lisp back up and insert the following:
(defun submit-reminder-form (widget)
(let ((new-reminder (create-reminder widget)))
(schedule new-reminder)
(persist-object *clockwork-store* new-reminder))
(reset-form-widget widget))
(defun create-reminder (form-data)
(with-form-values (subject summary) form-data
(let ((timestamps (get-timestamps form-data)))
(make-instance 'reminder
:emails (get-emails form-data)
:title subject
:summary summary
:timestamp (first timestamps)
:at (second timestamps)))))
Note that I wrote this in a way that's fairly natural to Lisp. I wrote submit-reminder-form in what almost resembles pseudocode and ensured it expressed my intent before worrying about writing a helper function to make that possible. So submit-reminder-form creates a new-reminder by passing the widget to create-reminder, schedules and saves that new-reminder in the Prevalence store and then resets the form. To make this possible, create-reminder uses with-form-values to scrape out the subject and summary from the form, then we grab the timestamps and emails using the functions we wrote for that earlier and instantiate the reminder object accordingly. At last the site is fully functional for sending email or text message reminders!
Recovering from Failure
We haven't covered what to do in case of an emergency reboot or other failure. Since everything is persisted by prevalence on submission the *clockwork-store* will still have our reminders. All we have to worry about is ensuring that all the timers get rescheduled. This is so simple it hurts. Reopen src/reminder.lisp and add the following code to the bottom of the file:
(defun recover-reminders ()
"A function to reschedule reminders after a reboot. Based on testing,
any that expired during the reboot will be sent when the schedule method is called.
Better late than never, right?"
(mapcar #'schedule (find-persistent-objects *clockwork-store* 'reminder)))
Calling
(recover-reminders) will schedule all the reminders in the store and whether Linux, SBCL or trivial-timers is to thank, a timer that's scheduled in the past will trigger immediately so you don't have to worry about some being lost during the reboot itself. Just add #:recover-reminders to the export list in the clockwork defpackage in clockwork.lisp and then call it after you load clockwork in your init.lisp file that runs when the server starts. Here's my init.lisp as an example.Next time...
At this point we've seen a tiny bit of the Store API and the default Prevalence store and learned a little about widgets and views. We still need to cover actions and navigation/dispatchers and it wouldn't hurt to demonstrate user registration and authentication as well as use of a SQL backend. Sooner or later we'll get around to all of those things.
Right now I'm working on a postmodern backend for weblocks and may also work on styling, polish, error handling and potentially a new feature or two for clockwork. Based on what gets done, the next article will either cover the beginning of a new project or continued improvements to clockwork.
Unless otherwise credited all material 
by Brit Butler 