Recent Posts

On Inaction

posted on 2013-03-04 16:28:00

Seeing Value

I've been experiencing the Dunning-Kruger effect a lot lately. At least, I've been feeling like a fraud. And while I could list reasons I'm not a great programmer, asking why I felt like a fraud has led me to something more interesting. I don't think I've worked at a company where I knew "where the money is coming from". What does that mean exactly?

All of this creates a surprising problem for me: The value I add is opaque from my perspective. I take it on the word of my superiors and peers that any value is present. This makes it essential that I trust and enjoy working with those people.

The Mythical Customer

It might not be immediately apparent why this is a problem. Find a company with decent people and culture and you don't need to be directly connected to the product or customers. Just churn out code and have fun. Advertisers will foot the bill. While it's true that you can sustain a business this way it certainly isn't ideal. The issue comes from just how decoupled the product becomes from the revenue. When it's time to grow revenue, you have to do it by attracting more eyeballs. Here's how that works:

  1. A Product Manager decides on new features or a UI overhaul to increase site traffic.
  2. Programmers implement those features with small tweaks and adjustments.
  3. The changes are released and traffic is measured for an increase. A good shop will use A/B testing to try and at least ground these decisions in data.
  4. Improved numbers are sent to advertisers to garner more customers and/or revenue.

But Product Managers are not users. Programmers are not users. Advertisers are not users. Sure, we use the product some to verify the code works during testing but we're not invested in it. A/B testing is not the same as user input. There is also no requirement that you correlate traffic with actual perceived value. Many companies just read traffic AS perceived value. Frankly, that's bullshit. Our loyalty is necessarily to the advertisers. They pay us...but the users are the real customers. The product just happens to be paid for by collecting data about how they use it.

So What?

Thus far, none of this should surprise anyone who has worked in the tech industry. Hell, this shouldn't surprise anyone with a Facebook account. It points however to a serious cultural problem in many tech companies: not letting (or demanding) your technical experts be, well, technical experts. A friend of mine calls this "{} for $". Many people rant about this as "Taylorism in software". It cannot be overstated that no programming paradigm nor software engineering methodology will eliminate the need to connect engineers to the product. Similarly, letting the engineers take the reigns is not anathema to good product design or improved value to the business. And this is not new. Quoth Don Eastwood in a 1972 Status Report on MIT's Incompatible Timesharing System:

"In general, the ITS system can be said to have been designer implemented and user designed. The problem of unrealistic software design is greatly diminished when the designer is the implementor. The implementor's ease in programming and pride in the result is increased when he, in an essential sense, is the designer. Features are less likely to turn out to be of low utility if users are their designers and they are less likely to be difficult to use if their designers are their users."

Earlier in the report, Eastwood says, The system has been incrementally developed almost continuously since its inception. Hello, agile kids. I'll say it again. Any company pretending that software engineering methodology or a given technology replaces the need to connect engineers with what they're building deserves to be skewered. The reason knowing "where the money is coming from" is so essential is that software is different than any other product in history. Because the design in a fundamental sense is the product. If you don't know what I'm talking about, I'd encourage you to watch Glenn Vanderburg's talk from RailsConf 2011. If your engineers don't understand the reason for what they're building, then the product can at best accidentally support the business. If you think you or your company can just scrape by for your whole career without getting eaten, I'd encourage you to reevaluate that assumption.

Finding Alternatives

One big reason most companies are hierarchies more concerned with maintaining market position than creating value is the inherent risk. Real growth comes from bets and empowering people to change what's needed to find "a better way" or "The Right Thing"...and that's terrifying. Few individuals or institutions have the guts, bravery, and stamina for continuous reinvention. It's exhausting. Not to mention that it puts the focus squarely on a company's employees. It seems our best chance at winning big comes from those kinds of risks though. Github and Valve's experiments in distributed management are a brilliant step in this direction.

In the Interim

A lot of what has had me feeling like a fraud is remembering how much I have to learn. Learning is a kind of reinvention itself though and one of the reasons I've loved computers since the beginning. So I'll keep learning and next time I'm in a programming interview, I look forward to asking the other hackers, Do you know where the money is coming from?

On Aaron Swartz

posted on 2013-01-13 12:36:00

I can't remember a death feeling this much like a personal attack. The best way I can describe the feeling is this: One of my heroes was slaughtered by my government for defending my beliefs. Of course, the sad reality is that Aaron took his own life, persecuted or not. As a disclaimer, I never knew him over the net or otherwise but I always looked up to him as one of our brightest lights. As written in another article, I have always aspired to be more like Aaron and can see myself winding up in his position, fighting his battles. That's probably why he is the most significant activist of this generation for me. To everyone else who is grieving, I love you. Hold together and we will create the better world that Aaron was so clearly fighting for.

EDIT: Initially, I had intended this post to explain in greater detail why Aaron's death has affected me as much as it has. I couldn't quite find the words but John Atkinson has done a beautiful job capturing my feelings on his blog. Please read it.

Other articles on Aaron's death follow:

Lessons from Coleslaw

posted on 2013-01-06 14:40:00

Is there anything more pointless than a new blog engine? Probably not. 4 months ago, I wouldn't have thought that I would be distracted from my Lisp 6502 emulator so long or that I'd have this much fun writing blogware. It is amazing, however, just how much you can do with a bit of time and ~600 lines of lisp.

Lately I've come to realize my favorite part of hobby programming is that I essentially treat it as creative writing. One of the reasons I love Lisp and find myself using it so much for hobby code is how easily it enables me to experiment with new coding styles. In Coleslaw's case, this has meant a stronger focus on CLOS and API design.

I like to think there's a clear stylistic shift in my projects, from the earlier and messy imperative of Paktahn, through the neat but overly macro-heavy cl-scrobbler, to the more balanced style of my present day code. It's no surprise that some of my favorite lisp luminaries, Peter Seibel and Luke Gorrie, talk a lot about code as literature and readable programs. Hopefully, I will continue to progress in that tradition.

Coleslaw 0.8 is hereby released. The biggest features are multi-site publishing and support for new content types. Here is an example bookmark or tweet-like content type that may ship in a future release, Shouts. See the NEWS for further details. It's time to get back to Memory Mappers for a bit and see if I can't get actual NES emulation going in pure Common Lisp. See you next time, Planet Lisp.

Welcome to the Future

posted on 2012-12-13 21:12:00

It's late 2012. I'm writing this in a gaming bar watching hopefuls for the next National Street Fighter Tournament practice. The FBI have lost track of a modern political dissident, a hacker vagabond by the name of Commander X. Last night I attended an escapist lightshow of a concert, where kids burned their brains out "like neon novas". Drones police the New York skies. My online handle is more crucial to my professional career than my "True Name". I drive a veritable supercar knowing that once self-driving cars are commercialized, classical driving could be a luxury if not slowly phased out altogether. I was born in the sea of information and raised by the sound of dial tones. The cyberpunk aesthetics of the 90s are quietly, imperceptibly interleaved into modern life. But like Alan Kay, I believe the computer revolution hasn't happened yet. My 10 year old self has everything he ever dreamed of. Time to come up with bigger dreams and see what's next.

Stalling in India

posted on 2012-10-16 18:10:15

I've been in India almost 10 days now and I'm still not really ready to say anything about it. I return October 20th. I am enjoying the trip immensely however. I've spent time trying to experience India, shopping and being a sightseeing tourist, reconnecting with old friends, and resolving a longstanding personal question. At some point there will be pictures but for now, since I'm having trouble finding my own words, here are some from my favorite writer:

"They are made of amorous dough. As soon as they turn twelve, love has begun to take them somewhere. They see its glowing torch from afar and follow it through the half-light of childhood..." - Carlo Gozzi, Memorie inutile

Elegy for Y.Z.

A year after your death, Y.Z.,
I flew from Houston to San Francisco
And remembered our meeting on Third Avenue
When we took such a liking to each other.
You told me then that as a child you had never seen a forest,
Only a brick wall outside a window,
And I felt sorry for you because
So much disinheritance is our portion.
If you were the king's daughter, you didn't know it.
No fatherland with a castle at the meeting of two rivers,
No procession in June in the blue smoke of incense.
You were humble and did not ask questoins.
You shrugged: who after all am I
To walk in splendor wearing a myrtle wreath?
Fleshly, woundable, pitiable, ironic,
You went with men casually, out of unconcern,
And smoked as if you were courting cancer.
I knew your dream: to have a home
With curtains and a flower to be watered in the morning.
That dream was to come true, to no avail.
And our past moment: the mating of birds
Without intent, reflection, nearly airborne
Over the splendor of autumn dogwoods and maples;
Even in our memory it left hardly a trace.
I am grateful, for I learned something from you,
Thought I haven't been able to capture it in words:
On this earth, where there is no palm and no scepter,
Under a sky that rolls up like a tent,
Some compassion for us people, some goodness
And, simply, tenderness, dear Y.Z.

On Visible Programming

posted on 2012-09-26 20:13:00

I have a bad feeling that I'm about to piss off a lot of people. Oh, well.

Bret Victor gave a very interesting talk at Strange Loop called "Visible Programming". From what I can tell, Bret is a very smart guy and an accomplished UI designer. I was surprised to find that while I agreed with many (all?) of his premises I disagreed with most of his argumentation. As I've received a question or two about it, I'll try to clarify my thoughts here.

I have three core points:

  1. His vision requires reflection and an editor is the wrong place for it.
  2. Many examples were domain-specific and/or trivial but he didn't talk about general editor extensibility.
  3. Much he suggests is already done or can be done with some solid effort and available technology.

On Relective Systems

Ironically, I gave a talk on a different but overlapping topic a few weeks back. As mentioned in Chris Granger's talk, examples like the ones he and Bret Victor gave are much easier when working with a dynamic runtime. Unfortunately, dynamic is conventionally interpreted as a language with dynamic types and, put simply, we should try to change that. I argue that a dynamic language is one that allows you to inspect values and update function and class definitions while the program runs. This is formally known as reflection. What I believe dynamic language advocates care about is the ability to work with a "live" system, not the presence or lack of static type checking.

Being able to visualize program execution means having a snapshot of the program state during each step of execution. Setting aside the impractical size of such a thing for programs with arbitrarily large working data sets, this requires either A) annotating every line or function call with logging statements/function tracing, or B) instrumenting the language runtime in some fashion to retrieve values during execution. That's why these examples are much easier with a reflective language like Javascript where you can hook arbitrary behavior into the Object prototype.

Trying to get an editor to do this for all languages is a fool's errand without a nice reflective runtime and API to retrieve data from. It's hard enough with that stuff! And that means we need reflective compilers and interpreters before we can have an ideal editor.

On Editor Extensibility

Many of Bret's examples were highly domain specific. While it makes sense to have a simple interface for toying with values in a visualization/drawing program, it's harder to see how to usefully apply that to something like protein folding software or even RSS feed parsers. Having editor extensibility in addition to reflective systems enables arbitrary widgets or modes for dealing with a given problem domain or data visualization need. Plain and simple, there's no way to build-in useful visualizations that are universally applicable. I admit I am not a designer or gifted at visualization so I may have simply struggled here.

What can be done today

Bret essentially wants to make the experience of programming more tractable through interactivity and tangibility. He suggested 5 key requirements:

  1. Enable the programmer to read the vocabulary.
    • Mouseovers/tool-tips on all source tokens is largely available today and seems to address this.
  2. Enable the programmer to follow the flow.
    • A visualization to step through execution could be written somewhat easily given something like Common Lisp's trace.
  3. Enable the programmer to see the state.
    • This is essentially a more elaborate visualization that requires a bit more trace data than the above.
  4. Enable the programmer to create by reacting.
    • Suggests that editor not only autocompletes function names but default values too so you can see their effect immediately.
    • Function name autocompletion exists in many IDEs. Pervasive default values are easier for typed than "untyped" languages. Still requires runtime support!
  5. Enable the programmer to create by abstracting.
    • Essentially demonstrated a refactoring system here. Keep feeling like I missed something key about his argument.

TL;DR

While philosophically well-formed, Bret seems to miss the fact that runtime support is required for a Visible Editing Experience to emerge. If the industry still doesn't understand that dynamism is about the runtime rather than types, clamoring for a magic editor will get us nowhere. I want to see a more interactive, tangible environment in the future as well but we cannot get there by arguing that IDE/editor writers need to step up their game. We need to make a concerted argument for the resurrection of highly reflective systems in both research and industry. Once systems with robust reflective capabilities are widespread, realizing a vision such as that described will be a week long hack rather than a decade-long million man-hour slog.

I'd also like to examine how far reflection can scale and a bit about making such a thing applicable to both novices and experts but I need more time to compose my thoughts so I'll save that for a future post. Any comments on this post or its mistakes, of course, are welcome.

Strange Loop Notes - Day 2

posted on 2012-09-25 10:17:00

Computing like the brain

SDR Properties:

  1. Similarity: shared bits = semantic similarity. A similar bit vector has similar semantics.

  2. Store and Compare: Store indices of active bits, don't traverse the whole thing. Subsampling is OK though!

  3. Probability shows errors are very unusual even with subsampling. If you do make a mistake, it's a close/semantically similar one.

  4. Union membership: Sets! Is this SDR (10..001) a member of this union of SDRs (00..001)? Very high correctness.

  5. The key to machine intelligence is sparse distributed representation.

Sequence memory:

Predictive Analytics Today:

Future of Machine Intelligence:

Behind the Mirror: The birth of Light Table

35 years later...

Runaway Complexity in Big Data

Missed due to chatting with Chris Granger and other awesome folks.

Computer Architecture from the 1960's

Guess Lazily: Making a Program Guess and Guess Well

Okay, I couldn't keep up and had you been there you wouldn't blame me.

But this was cool and analagous to the Byrd/Friedman running programs backwards and forward stuff from yestereday.

Also, Buyer Beware: Oleg's papers may be more clear than his presentation style. :P

An Audobon Society for Partial Failures

Expressing Abstraction, Abstracting Expression

Aspects of Expressiveness:

Expressiveness over performance, every time! (in Ioke)

Types of Abstraction:

Macros!

Different kinds:

  1. C-style Preprocessor macros. No structure. Just string processing! AAAAAGHHHH.
  2. AST macros. Work on S-Expression, not AST.
  3. Same language available as transformation language and host language. Metalanguage, etc.
  4. Often use functions in a complex macro.
  5. Template macros. Type-directed expansion, turing complete.
  6. This is scary because they are TWO different languages.
  7. Similar situation to complex Type Systems in Haskell/Scala.

Modularity!

Static typing!

Generics!

Lin-guis-tics:

Communicating indirectly with many audiences though. The machine, team members, ops, etc. Even business stakeholders! Totally unique to programming.

Syn-tax:

Visible Programming

5 Principles to aim for:

  1. Must enable the programmer to read the vocabulary (program).
  2. Must enable the programmer to follow the flow (Call-flow-graph).
  3. Must enable the programmer to see the state (as it executes).
  4. Must enable the programmer to create by reacting (iterative, interactive development).
  5. Must enable the programmer to create by abstracting (what sort of tools?).

Examples of these principles in a hypothetical environment: (JS & Processing for this talk)

  1. Normally we go to the docs or source. This sounds like using 4 sliders to choose arguments to something in Photoshop. UI garbage.
    • Mouseovers/tool-tips on each token in the code. Make meaning transparent!
  2. Great explanations show, not tell. Make it possible to see the output of a given form as inputs change. Explain in context!
    • Imagine a cooking show that introduces ingredients and then cuts to the result. You can do it yourself!
    • Step through execution via a slider. This helps but it is not enough. We can't see patterns in execution.
    • Plot it on a timeline. Which lines executed when? what patterns or alternate executions could there be?
  3. It is common to expect programmers to manipulate code in their heads. WHY?
    • It is the responsiblity of the environment to show what changes a line causes in the running program.
    • Show what lines executed, when, and what effect they produced.
    • Larger programs just become a data visualization problem.
    • Show the data, show the comparisons, eliminate hidden state.
    • Making global variable change visible is an option for inherently stateful lines.
    • More programs running forward and back. (editors note: you guys are jerks and you hurt my heart)
  4. Composing things in your head doesn't scale. Maybe this is why only small programs are beautiful! Use the environment as an external imagination.
    • Live coding is one way to approach this. Bit of a straw man argument here.
    • Core idea is you want to have program supply you ideas, not just keep you from hitting recompile.
    • Take completion further, what if autocompletion lists also included suggested values?
    • Useful but clearly limited to the domain of this specific problem: Processing/GUIs and visualization.
    • Dump all the parts on the floor so you see your raw materials. Functions are lego blocks. Have API browsing. Offers another problem-specific example.
  5. Encourage starting with constants, than adding variables and functions later once you know what behavior you want.
    • Editor can help by offering "factoring relations".
    • I know this wasn't intended to resemble a working production system but I was hoping for more than this.
    • I thoroughly agree with his premises but I do not think this was a convincing way to demonstrate their need.
  6. How does this scale to real-world programming? I have answers but I think it's better to question the question.

  7. Asking how to scale to everyday programming is like asking how engines benefit horses. We have to work like this.

  8. Argues against static runtimes. I agree but this was a weak treatment. There is tooling to work around this.

  9. "There is no future in destroy-the-world programming. It's got to go." - Agreed.

The State of Javascript

Apparently all that was library and application improvements. What about code generators?

So bytecode...

Epilogue

I was going to have to miss the last keynote and two talks as well as all the camaraderie that would occur after. Delta had screwed up my flight and I didn't have a hotel room for the night. Thankfully, some Loopers came to my rescue though Delta still managed to extract $170 from me. Soulless corporations! It was totally worth it. I can't think of a more worthwhile or energizing way to spend 3 days. I'd like to thank everyone I was able to spend time drinking and chatting with especially Scott Vokes, Paul Snively, Andreas Fuchs. Brian Rice, and Jose Valim. Obviously, I never want to go home.

Strange Loop Notes - Day 1

posted on 2012-09-24 10:46:00

ELC/Preconference talks

Though I didn't take notes on these, or today's keynotes, I have seen quite good coverage of ELC and Strange Loop talks here.

Potificating Quantification

Functional Design Patterns

A Whole New World

Possibly the talk I'm most excited for today. DESTROY ALL SOFTWARE!

3 Confessions

  1. Wrote "An Editor"
    • Modal, Terminal Only, Neither VIM nor IDE
    • Layers! Annotations on source. Diffs. Tracebacks from prod logs. YES!
      • Crash - Have to parse logs+traceback, maybe use different checkout.
    • Interactions - On-demand Class Hierarchy/CFG. Code navigation methods.
      • No static analysis! Language-specific FIFO queues for program traces! Fork+render with graphviz. FUCK YEAHHHHH!
    • Answer questions like: What code does a web request hit? More importantly, what code might have reached this crash point in our traceback?
  2. Wrote "An Terminal"
    • DEC VT100 has determined terminal protocols for 30 YEARS. Powered by an 8080. @1978.
    • Add raster graphics, 24-bit color, momentary keypresses, font styles.
    • Use for more editor layers! Tag lines with profiling info. Bottom 95% grey, others yellow or red. Same thing for Type annotation. Record traces, remember?
    • Do you want it?
  3. Wrote "An Lies". HAS BEEN LYING.
    • All bullshit. C-c f t, flip all the tables.
    • Takes a long time to fake all that shit.
    • "Ship often. Ship lousy stuff, but ship. Ship constantly." -- bullshit
      • I KNOW that all software sucks.
    • Legacy & Paralysis? Legacy == Paralysis!

They will not merge our kernel patches. How do we move forward? Our "Shipping Culture" is poisonous to infrastructure. We just accrete low level infrastructure. Programmer Archaeologists are we. INCREMENTAL DEVELOPMENT WILL NOT WORK FOR THIS.

Type-Driven Functional Design

Clojurescript

Data Structures: The Code that isn't there

"A Data Structure is just a stupid programming language." - Bill Gosper

Perhaps instead...

"A data structure is just a tiny virtual machine." - Scott Vokes

"The cheapest, fastest, and most reliable components are those that aren't there." - Gordon Bell

The Database as a Value & Making Javascript Fast

Elided due to battery life.

Now with even more Lisp!

posted on 2012-09-19 20:47:00

I haven't finished my headless NES emulator in time for Strange Loop. On the other hand, I have done some cool things that I didn't anticipate. Here's what's been going on in hackland lately:

Coleslaw: Heroku & Plugins

The majority of my efforts have been related to my new blogging engine, Coleslaw. I've substantially cleaned up the rendering pass, added optional RSS feeds for specific tags, revamped the plugin architecture, and added a plugin for Disqus support. Jose Pereira also wrote a heroku buildpack for running Coleslaw so if you feel like having a simple managed install, problem solved!

While docs are still needed, here is a rough overview of the plugin architecture. I'll add a simple, hello-world-ish example to the README shortly.

Colorize Reborn

I am pleased to announce I've taken over as maintainer of colorize. It provides the syntax highlighting in Coleslaw's 3bmd markdown mode. I've backported patches from lisppaste to colorize for Haskell, Erlang, Python, and a number of other languages. I also added very rudimentary support for Clojure highlighting. While I'm interested in seeing further language support (particularly Clojure and Rust) I lack the time to work on further coloring modes myself. If you have any inclination to hack on colorize to add support for new languages or otherwise improve it please feel free to contact me. I'm more than happy to review and merge pull requests. :)

Wanting Types, Demanding Mirrors

I delivered a final Lunch and Learn at CMGdigital on Dynamic Systems. I have a screencast of it like my last talk but I haven't uploaded it yet. I'm a bit more out of my depth this time and am still considering tweaking the content and rerecording. The slides are linked above though and I would welcome comments. I'm also working on a Magic the Gathering tutorial/unsession for Strange Loop and an Emacs Crash Course for my new coworkers at Primedia. Finally, I'm giving a talk on the internals of cl-6502 to the Atlanta Lisp User Group on October 1st.

Vacietis and trivial-dump-core

Vladimir Sedach has been working on a C to Common Lisp compiler called Vacietis for a while now. It's become mature enough to generate Lisp executables for small C programs. However, dumping executables isn't a portable CL feature so I contributed a minor pull request to use trivial-dump-core to improve portability from ccl to clisp, sbcl, and ccl. I hope trivial-dump-core sees more use and gains support for more implementations as portable executable creation would be a nice thing to have.

Swanky Developments

posted on 2012-09-04 13:22:05

Now that the i's have been dotted and the t's crossed I'm pleased to announce I've accepted a new job. Starting September 17th, I'll be a Senior Developer working for Primedia. I'll be helping them migrate from ruby to clojure. I've been meaning to spend more time hacking Clojure as it is. I'm particularly delighted that I'll be in something of a teaching role and able to share my knowledge and experience with lisp with interested hackers.

CMGdigital has been a phenomenal place to work for the last year and I'll miss everyone there dearly. I wasn't looking for a new job but Primedia found me and this was in many ways the right opportunity at the right time.

I'm also very excited for the arrival of Leiningen 2.0 and happily running the latest preview. After using quicklisp, I disliked having to setup a mock project to experiment with arbitrary libraries in Clojure. Leiningen 2 uses a library called pomegranate under the covers which allows modifying the REPL classpath. Thus, dependencies can be easily added to a running REPL and experimented with!

In other lispy news, the dream of endless swank backends and SLIME on everything has died. Previously, I had coerced SLIME into running Clojure, Scheme, and Common Lisp simultaneously and knowing which filetypes to associate with which repls. It took a lot of fiddling. I actually had a rant against the proliferation of swank backends for other languages like Chicken Scheme and Clojure happening outside the main SLIME tree. Anyway, between Emacs 24 shipping package.el, marmalade, nrepl.el for Clojure, and Geiser for scheme, the situation has resolved itself even if the infinite SLIME dream is dead. And ultimately, that's better for hackers everywhere...so who am I to complain? :)

PreviousNext