Reboot

posted on 2021-08-30 16:07

This has been an incredibly full and difficult year. Norma and I got vaccinated. Her mother had a serious and ongoing health crisis for several months. During that period, we both had unusually busy quarters at work. I applied for a promotion at the encouragement of both managers and peers and narrowly missed it. My product manager got hired away to be a CPO and several engineers left the product section I work in. As a result, the squad I started is being re-absorbed into the other squads in our product group. Management hopes to restart it at a later date. Finally, the long discussed garage renovation on our home is completed. Now it functions as a climate controlled office and synth dungeon.

At the end of all this, I'm deeply exhausted. I'm very thankful that lately I have been able to enjoy much more human contact thanks to being vaccinated. Seeing friends in person and giving them hugs goes a long, long way. I'm trying to get back in the groove on pursuing creative projects but it's very hard.

Lately I've been thinking about how if I don't choose to enjoy my days, then I am likely to stress myself out. Work can't make me happy, relationships can't make me happy. I care deeply about relationships and community but I have to choose to enjoy my own experience. I have to choose to validate who and where I am. I deserve to feel good. If I don't choose to let myself feel good, and pursue the things that excite me without judgement, then I will find a reason to be hard on myself or stress myself out.

I've been having to remind myself that neither money nor work nor relationships are inhibiting my ability to lead the life I want. It's all me at this point. I just have to start living more fearlessly and letting myself be me in the way that feels right. Damn the consequences. Something to keep working on. Step by step.

Feeling Stuck

posted on 2021-03-07 22:15:00

It's been a year and I'm still thinking about Halt and Catch Fire. I haven't started my 3rd rewatch but it'll probably happen soon. I wrote a bunch of toots about it recently but it may not capture how I feel as well as this infovore article.

The reality is, I love the show because it's about people learning to have healthy relationships with work and to love themselves. I tend to form great bonds with coworkers but my relationship with the work itself and my ability to love myself is just damned fraught.

I feel unbelievably drained lately. I'm in a weird place at Calendly where I'm sort of in between being an IC and a Manager. I'm convinced I'm doing less work than ever while receiving good reviews and "Top Performer" recognition. I just find it insanely hard to praise myself. I've thought a lot about it and one reason I've struggled so much (and sought a Team Lead position) is that I don't know how to praise myself for writing software.

I had to think really hard about that to realize that I don't like much software. People use computers in all these ways that just don't make sense to me. The software I get the most excited about is small and personal. An emacs configuration. A 1000 line blog engine in Lisp. A handcrafted website conveying intimacy and joy.

It's okay that professional software isn't about that. But I have a very hard time figuring out what the "best way" to architect large systems is because I don't want them to exist in the first place. I can speak up when I see an obvious way to improve what we deliver but I actively avoid ownership because it's not the software I want to see in the world. Or at least that I want to use.

That's left me in a tough place trying to figure out what I want to do next professionally but it's been an important step. Maybe moving to a more managerial side will be good. I'm also aware that I am much more able to praise myself for my efforts as a teacher than as an engineer. Teaching is, in many ways, more congruent with my values. For better or worse, teaching examples are usually small applications that can eschew most of the thorny aspects of production software.

It also hasn't helped that I've struggled to commit to hobby coding. I've struggled with depression a lot throughout the pandemic. I get a ton of my energy from my sense of community and sharing experiences with close friends. It's been very difficult to keep a good perspective stuck in the house all the time. I'm also aware that coding all day has taken a lot of the desire to joyfully explore a computer in the evenings away from me even if I did have collaborators handy. I turned a hobby into a career and I definitely have some regrets about it lately.

One thing I have enjoyed in the last few years is listening to a lot of electronic music and starting to dabble with synthesizers myself. With Norma and I both working from home though, the former pseudo-studio space has become a home office. We started remodeling the garage to be a hobby/office space last November, but that work still isn't done.

So here I am. Coming up on the 12 year anniversary of Dad's death, the 10 year anniversary of my entry into the workforce, and my 35th birthday. I'm having a hard time forming healthy habits for managing my stress and not sure what I want to do next. I don't feel I can complain because, well, I'm extremely well off! Calendly is doing well and treating me quite well, I'm safe and at home, in a supportive relationship, my family and friends are safe.

And yet, the urge to pull the plug and change my circumstances is so powerful. I might consider it if only I was certain about what I thought should come next.

Heaven is a Place

posted on 2020-04-14 14:35:00

NIM

Songs of the Day:

• Pixies - Velouria
• The Sea and Cake - Four Corners
• Blood Orange - You're Not Good Enough
• Soccer Mommy - Circle the Drain
• Wild Nothing - Midnight Song

It's been a bit of a week. I'm on vacation from work, thank goodness. I had really run out of steam on my projects. It was difficult to focus and I was berating myself a lot which only makes things more exhausting.

There's so much I want to work on.

I have a really, really hard time making myself happy. In a lot of ways, I think I don't know how to play by myself. You would think as an only child that it would come naturally, but it really doesn't. I mean, I can list things I think I'd like to do: read books, play video games, write code, etc. I just struggle to do any of those things or feel good when I actually do them. I think this is why the quarantine has reminded me of what summer break felt like as a kid. There was an initial elation at this sudden freedom in your schedule ... and then a gradual despair as nothing seems to matter without someone to witness it.

I've been rewatching Halt and Catch Fire and very emotionally attached to it. I think one of the reasons is that it's about these characters who obsess and get fixated on projects but really struggle in their relationships. I feel like I'm pretty happy with my relationships but hate myself for not moving forward on the projects I fixate on. I identify with multiple characters on the show and I think I struggle emotionally because they wind up alone, both romantically and in terms of collaborators. In many ways, the show is about failure and how the characters deal with it. And after seeing them grow as people, work so hard, and love so fervently, it's heartbreaking to me to see that failure.

I've been struggling with a need for external validation my whole life. I think there are a few components to that:

1. I don't really trust my own evaluation of myself and my contributions. This is mirrored in HCF when Joe asks Cameron "What is it that I do?". He honestly can't convince himself that his contributions make people's lives better rather than worse.
2. Because I get very fixated on what interests me or how I think things should be done, I need to have a fair amount of autonomy or executive function to be happy and have fun. But if I'm doing anything (project or recreation) purely alone I lose steam and belief that it's worthwhile. The need for both structure and validation from collaborators while choosing not only the activity but also most of how it's carried out makes it really hard for me to find people that will come along on adventures with me.

Two different relationships in my life stand out as being unusually good for me by playing to these challenges.

1. The Iron Yard was probably the happiest I've been at a job ever. I had a tremendous amount of freedom and responsbility in how I ran my classroom. But thanks to the other staff and the students, I never really felt like I was "on my own". I rarely got into that crazy "does it matter, is it really worth anything" headspace. I had external validation (mostly students), collaboration and friendship (staff), and autonomy in how goals were pursued. There was some tension because I sometimes wished I could change the broader educational goals or thought we weren't honest enough with our students about the challenges they would face, but this was dwarfed by the rest.

   I think this is one of the reasons teaching has often been a good job fit for me. Quoth the showrunners from Halt, "I think teaching is a way to spread your love of a thing without needing to be the victor in that particular arena." In a lot of ways, I struggle to to be an engineer instead of a teacher because I either genuinely don't think the technical problems at the company are interesting or because I think I need to be the victor to prove my worth. In teaching, the problems aren't the important thing, getting people to connect with themselves and their curiosity is most important. Tech is just the setting. That feels almost perfect to me.

2. Ben Minor was my roommate both in college and after and is one of my dearest friends. Ben is a very easygoing guy and happy to do most whatever you want to do all the time. It helps that we have some overlapping interests (aside from code) but in 15 years of friendship, he's rarely not been open to doing whatever I feel interested in doing.

   I often have a hard time working on things on my own. Even if I'm legitimately interested in them, I struggle to make myself believe that they matter or generate forward motion pursuing them. I desperately want collaborators. But frankly, I'm pretty difficult to collaborate with because I have both pretty specific ideas about what we should do and how we should go about it. This makes me relate to Cameron from Halt and Catch Fire because she is genuinely terrible at working with others even though she would love to. She needs to be the special brilliant child a bit too much.

   But this also extends to how I relax and have fun. In an interview with the actor who plays Joe Macmillan, he said: "He wants his friends to play his game. I guess that's his flaw. He wants to be in the sandbox with his friends, but he wants his friends to be building the same sandcastle."

I don't know how to fix this, or even how to work on it really. Norma mentioned recently that when something makes me unhappy my reaction is to stare at it until I feel I understand or can move past it rather than to be avoidant. It results in a lot of unhealthy feeling "stuck time" where I feel anxious and spiral into my unhappiness. Hopefully one day I can find a little more peace, knowing that I'm valued in the world for loving people in spite of being a little hard to work with. I'm still a bit worried that I'm not a great software engineer because I'm more interested in studying, learning, and sharing, than building things. But one thing at a time.

So sure, this week I would love to:

• Work through Crafting Interpreters
• Rewrite the graphics layer in Clones
• Work through more On Lisp, SICP, CS:APP, or Algorithms
• Investigate tools for literate programming / publishing programs
• Start using org-mode
• Make a great mixtape

But I have to take care of myself first. And today, that might just mean doing laundry, missing dad (and wishing I could listen to music with him), cuddling the dogs, and writing. Maybe I'll spend some time in the hammock. All for now...

In Praise of Unfocus

posted on 2020-02-01 22:45:00

De-emphasizing output

I often tend to judge myself, at work and outside of it, on my output. Old habits are hard to escape. But there are a lot of reasons this is an instinct to be resisted. First of all, if you're tackling suitably interesting projects it may be hard to predict progress or even define good stopgap goals. Many people experience cultural pressure to measure themselves in terms of their output or to be efficient in all things they pursue. That distortion in what we value often leads to a lopsidedness in our goals and pursuits. There are two defenses that have stuck with me lately against such ideas. So I want to tell you about them in the context of last Sunday when I was unfocused and then anxious about having spent my time that way.

Breadth over Depth

In a single Sunday I got excited about: 90s techno (Tri Repetae, Hard Normal Daddy), new techno (Djrum, HVL, John Talabot), Smash Bros Melee both for its depth to competitors and my choice to be only a semi-serious player, a study group I started at work to go through Crafting Interpreters, making music with Modular synths, making music with Renoise instead, Open hardware efforts like Pine Phone and MNT Reform, Grammatech's open source SEL project for doing Binary Analysis and modification with Common Lisp, InterimOS, and a slew of other odds and ends.

Even if these cluster around nerdy themes, that's an awful lot to enjoy in a single day. It's easy to lament that nothing was produced, that there is no output to share, but that ignores the fact that being plugged in to so many things often means forgoing total immersion in a single piece of subject matter. And there are network externalities at work too. Discovering music means I can play it next time I co-host The Mobius on GA Tech radio with my buddy Matt. Being aware of software and hardware projects means I can share resources with other hackers.

People over Performance

But all that wasn't enough to keep me from worrying that I don't focus enough. Or I focus on the wrong things. I had a performance review coming up at work so that might have had something to do with it. Even though the company has a relaxed culture, I'm a worryer and a catastrophizer by nature. I was getting worked up over the fact that I don't ship as much as some other members of my team. I was trying to imagine exactly where the bar was and what objective, quantitative measures would be used to evaluate my performance. I was thinking about how I had never felt like I was the developer doing the most challenging or glamorous work at my past jobs and I still had a slower rate of shipping tickets than those taking on that challenging work.

After a little while, I thought especially about Emcien. Emcien was a great opportunity for me and a cool Ruby/Rails shop to work at. I was there from early 2013 to late 2014. More so than other jobs I've held, I think of things at Emcien as having gone sour even though I left in decent terms. I got into a negative headspace about the value of my work and its perception in early 2014 and that re-inforced until I was dragging my feet constantly. I wound up putting in my notice because I was unhappy, but the work environment didn't do it, I did it to myself. So, did it end in failure?

Here's the really interesting part. After I left that job, where I assumed I was a waste of money and possibly resented for not being a bigger contributor, I wound up becoming the best man of one of the engineers and a good friend and confidant of another. There were 5 devs total, including me. The 2 jobs I got immediately following Emcien were working with those two individuals and they both had positive things to say about me during the interview process.

Probing for Answers

How do we explain this? There are a few possibilities I can think of. One is that I actually was just as productive as everyone else. Another is that even if I was less productive I was working on things others were not excited to work on and that freed them to do what they were passionate about. But the explanation I like is that while I shipped less code than my peers, I made sure they felt great every day. As much as I was able, I shared energy with them and supported them in their efforts while still making progress on my own tasks, even if a bit slowly. I just loved on them.

It's easy in software to get fixated on people working on the most impressive or high profile projects. I can get tunnel vision thinking about career growth as only coming through huge expertise. But even as a software engineer you can be as valuable for how you support others as for your individual efforts. It could be more important to your career goal to be energizing and well liked than productive. Which isn't to say you should go be popular and ignore your work, just that focusing only on output distracts from a more realistic view of how people function as part of a team.

I sometimes struggle with my instinct to be unfocused, or am ashamed I'll be thought of as a dilettante or dabbler, as if the only respectable option is to be myopically focused on one area of knowledge. To have one project of grand scope. I need to remember, and I imagine I'm not alone, that embracing my varied interests both nourishes me and is a valid choice that can enrich the communities I particpate in.

And that's enough of my soapbox for one day. Happy Hacking.

Five, Seven, and Ten

posted on 2020-01-05 14:30:00

10 years ago, I was studying CompSci at SPSU. I had just gotten my first smartphone, a Nexus One. I had started contributing to my first open source project, Paktahn. Dad had died about a half year earlier.

7 years ago, I had just met a woman I would fall in love with, Norma Nyhoff. I was living on my own for the first time at Arts Center Tower in Midtown. I was a Senior Engineer at Rentpath and not very happy about it. I had written and open sourced Coleslaw.

5 years ago, I taught my first class at The Iron Yard, a bootcamp where I helped over 100 people become programmers. I was living with Norma and our two dogs. We were about to rent a house in East Atlanta.

Last year, I got married. I taught another 100 people to program at the Flatiron School. I joined Calendly as a Senior Engineer. I made substantial progress on rawbones, an NES emulator written in ReasonML with one of my best friends, James Dabbs.

Somewhere I read that "Unattainable Earth", the title of my favorite Milosz book, is most closely translated as "Earth too huge to be grasped". In a similar way, these events feel like they surpass my understanding. In the last decade, a life has grown. I foolishly tend to think of my life as something that is planned or designed, but that is not what happens to us. We take a step and a new world blooms as our feet land.

I hope I can be excited and curious more than fearful in 2020. I hope to write more here and perhaps listen to myself again, the way I used to.

I came up with some resolutions but won't share them. The important thing is the shift in focus. I want to pursue programming for artistic purposes, generative graphics and sound. And maybe learn to solder and build a keyboard. Hardware seems fun, graphics and audio seem fun. The web is powerful but not exciting. So I'll work, spend time with the people I love, and see where my passions lead me.

Research Goals

posted on 2018-08-22 12:59:00

A Recent Dilemma

Lately, I've been thinking about what I want my next job to be and I've been strongly tempted to pursue grad school. I only fell further down the rabbit hole when I read this tweet by Tony Garnock-Jones:

Reading things teaches people how to write. Analogous, if we are to place programming at the same fundamental level, using a program should teach how it works. But we don't see this.

— Tony Garnock-Jones (@leastfixedpoint) August 6, 2018

This problem is very close to my heart. Yet I'm ill equipped at present to tackle it. That suggests I should pursue a PhD to get better tools but I have some serious concerns about that (above and beyond selling my house). Let me explain.

The State of CS Research

Thinking about CS academics, it seems that most research aims to support building larger systems by improving software:

1. Correctness, or
2. Performance

I feel really weird because neither of those things interest me. Sure, modern software has plenty of bugs and it could always be faster. But software is eating the world anyway.

My Concern

Software is the fastest growing store of "how to" knowledge on earth and is mostly inaccessible, not only to the general population but programmers too.

What do I mean? Well...

There's no such thing as code being "readable", not only because being readable implies assumptions about the context of the reader but also because most programs cannot present a linear narrative! As a consequence code cannot be self-documenting. We should strive to write clear code but suggesting that code is its own documentation is untenable.

I started working on a Nintendo Emulator in Lisp to explore the idea of readable code and see if I could make the emulator source a good pedagogical example of what a computer does. I failed.

My Goals

Now my primary motivation working on the emulator is finding ways to generate an explanation of binaries without just recovering the disassembly. It's the intent and constraints that matter, the shape of the problem, not necessarily the solution the developers wound up with.

Indeed, if we could recover disassembly or even get the original source out of binaries (compilation being an inherently lossy process) that would only get us more code that we would need to comprehend. I.e. More how-to knowledge that isn't readily accessible. Legacy code and software preservation only makes this need more urgent.

I should note that I don't think all software needs to be preserved. I just think it's a travesty that we're 60 years into the project of programming and our tools for asking computers about how programs behave are so poor and so specialized.

Computers could be much greater tools for knowledge dissemination than they presently are because they can execute models. We currently use them to disseminate static knowledge (web pages, PDFs, videos) instead of executable knowledge.

I continue to want to find a way explain code without relying on static methods like documentation, the code itself, or even types and tests. I dream of better tools for communicating and reasoning about the work software systems do than source code.

Putting it in the simplest terms possible:

When I was 8, I desperately wished I could ask my family computer, "Wow! How did you do that?" I still can't today and I spend a lot of time thinking about what a solution should look like.

Footnotes

see also: Peter Seibel and Akkartik, though Luke Gorrie may be a counterpoint

Things I learned writing Clones

posted on 2018-07-29 12:08:00

I'm turning 32 in a week so thank goodness I'm finally making progress on clones. After my last post, I didn't work on clones for 7 months. Then in May, I just sat down and started hacking. Despite some gaps, there has been steady progress.

There's still a lot I want to do and audio isn't implemented so that's next, but for now I'm going to try to summarize the current status and some of the lessons I've learned thus far.

Current Status

Clones, At Present

The Clones CPU emulation is finished and tested and there is support for input handling and basic graphics support (backgrounds and sprites, scrolling is next). A lot of what determines compatibility for a NES emulator comes down to mapper (cartridge) support and the accuracy of the PPU support. In that regard, clones supports NROM, UNROM, and MMC1 though UNROM and MMC1 have some issues that need ironing out once scrolling is finished.

The circuit board used in NES cartridges actually varied and added additional capabilities to the console, primarily a paging system for switching banks in and out of memory to allow for larger levels, more artwork, more game code, etc. The different cartridge types were called mappers. Thankfully, 6 different mappers accounted for something like 80% of all games commercially available in the US. As a result, mapper support is a big deal since you can't play a game without the matching cartridge support.

Clones, In the Future

The first priority is fixing some sprite glitches and getting scrolling implemented. Once that's done, fixing up the lingering issues with UNROM and MMC1 will take precedence. Once Mega Man 2 is booting, then I'll start work on the audio.

After that's done the real fun begins. I have all sorts of ideas and ambitions for how to build a Control Flow Graph of the game dynamically while it executes and then let the player annotate the structure and save it for later revision. I want to be able to reverse engineer old games interactively and am wondering how much the computer can help in the process with the use of Constraint Logic Programming tools like screamer. In general, I'm interested in how we can examine shipped binaries at runtime as a teaching tool for how the software and hardware work.

More on this soon, I hope. 🙏

Lessons Learned

Test ROMs == Joy

There are many test roms for ensuring that various components in your NES behave accurately. I found it particularly useful to write the memory interface, addressing modes, disassembler, and a stepper for the CPU with no instructions implemented. Then I had a unit test which looped over a verified correct log for a ROM called "nestest" which exhaustively checks the operation of all legal CPU instructions. After I could run the test until it failed, implement a single instruction, and re-run. I had all 56 instructions with their various opcodes written in a day. Super pleasant!

It helped that I'd written a CPU emulator before, of course. This process required having a good idea up front about how I wanted to interact with memory, represent addressing modes, and execute instructions. If you don't understand those pieces though, you'll run headlong into them while trying to implement an emulator anyhow so start there. Spend some time reading nesdev wiki or asking questions online if you need to. 🤘

A Good Macro-Friendly Decomposition for Instructions

There is a bit of a Gordian Knot in the CPU in how the addressing modes and different opcodes interact if you want to define each instruction exactly one time without a mess of switch statements for the different variations. In short:

1. Addressing modes should only access CPU and Memory to compute an address. Any cycle counting (e.g. for crossing pages) can be done at the call site with macros!

2. Your opcode-defining macro should set up address and argument variables, or an update function as needed based on the access pattern of the instruction.
   This has bitten me on previous attempts as I assumed the access pattern came from the addressing mode rather than the instruction itself. Instructions can be implied and use no argument, or only use the address and jump to it, or read an argument from the address, or write a value to an address, or read a value, modify it and write it back. It was very worthwhile to split these cases out and handle them independently. It meant a little extra work while writing up the metadata but kept concerns separated later.

3. Separating the opcode metadata from the actual instruction definition. This is more arbitrary than the earlier recommendations but it felt very clean while hacking the opcode definitions and I think I only found myself going back to edit the instruction metadata one time from making a typo.

PPUs == Pain

PPU stands for Picture Processing Unit and it was the graphics card in the original NES. The central innovation of the PPU was that it supported pixel-level scrolling of levels.

I have no experience in graphics or game programming so this was a big challenge for me. Four other factors contributed to the difficulty of writing the PPU:

1. There isn't really a suggested ordering of tasks like there is for writing a CPU emulator and subtasks are hard to test in isolation.
2. The data layout of the PPU is complex: OAM, nametables, pattern tables, attribute tables, etc.
3. The test ROMs that exist generally assume you have the basics working.
4. While many working open source emulators exist, the control flow was difficult to follow without getting lost in the weeds and it was unclear why different calculations were used.

I'll try to tackle these briefly and write up more details at a later date.

A Suggested PPU Ordering

First, you need an object to represent the hardware state. It'll need to access the currently loaded game ROM for graphics data so remember to give it a slot for storing the cartridge object.

Second, you'll need to implement the PPU memory map. There's no operating system on the NES so there are no video card drivers and you'll do everything yourself via Memory Mapped I/O. If you've never heard of memory mapped I/O, the idea is that reading and writing to specific addresses in memory directly manipulates the PPU so write those methods and wire it up!

Third, you'll want to get the timing synchronized between the CPU and PPU. You'll want to do this before trying to render graphics probably as many games wait for an interrupt called vblank from the PPU that the graphics card is ready before even reaching the title. Many games will infinite loop until the PPU wakes them up with this interrupt, then do the work needed to render the next frame and return to the infinite loop. This is part of why it's so important to get the timing right.

Fourth, you'll want to make sure the address computations are right. This was the single hardest bit of code for me to get right in the PPU. It's also the code I'm happiest with and hoping to figure out how to test in an automated way for next time.

Fifth, try to just render the backgrounds using the addressing logic you arrived at ealier. If you can get backgrounds rendering correctly, you should be well on your way to getting sprites and scrolling working. With any luck, the PPU operation should start becoming clearer.

PPU Data Flow

Internally the graphics are represented as 8x8 tiles that are either sprites or backgrounds. Crucially, the information needed to render those tiles is divided up into the different areas inside the PPU: nametables, attribute tables, the palette, and pattern tables (in the ROM).

Nametables represent the background and are 960 byte long arrays where each byte is an index into the pattern table for an 8x8 tile. Why 960 bytes you ask? Because the NES resolution is 256 by 240 and if you divide that by 8 (pixels in a tile) you get 32 x 30. 32 * 30 = 960.

So nametables point to the "pattern" or texture that will be used for a given tile but for space reasons that pattern doesn't actually store all the information about what color it should be. The pattern table is 4kb and holds 256 tiles with each 8x8 tile taking 16 bytes to store. Those 16 bytes are enough for each pixel to get two bits to represent a color ... so 4 options.

The PPU has a 64 byte palette table to select 32 colors for the background and 32 colors for the sprites. But why bother when each pixel in a pattern can only count from 0-3? Well, did it seem a little odd that the Nametable was 960 bytes? That's because the last 64 bytes in that kilobyte are used to store something called an attribute table. Every 16 tiles share a single attribute byte which determines the top 2 bits of the palette index for tiles in groups of 4. There are implications from this about how many colors can be represented in a 16x16 pixel area of the screen, on a scanline, etc.

It's pretty confusing until you sit down and draw it all out. A lot of the calculations for the PPU are exactly this sort of thing. You can just imagine the hardware designers saying: "But how do we do it with less RAM to bring the price down?"

This has been written up well elsewhere, Scott Ferguson's blog comes to mind. But I still never found a high level description of how the PPU renders that wasn't based on perfectly emulating the state of a bunch of internal shift registers and latches and running the PPU cycle by cycle. And, pardon my french, but that's fucking gross. Not because it's inaccurate or slow or anything like that but just because it's hard to see the forest for the trees.

Here's something like how I think of background rendering now. Sprites are more complicated but follow the same basic framework:

1. For each scanline, you have to draw 32 tiles.
2. For a (background) tile, you have to get the nametable byte and attribute byte.
3. The pattern table has 16 bytes for a tile but we only need two for the 8 pixels on a scanline.
4. Go ahead and extract the two high bits of the palette index from the attribute byte.
5. Now loop for pixels 0 to 7 and combine those bits with the bits in the pattern to get a color.
6. Show the backdrop color if it's zero (transparent), otherwise show the background color.

I know this is inaccurate, but it's clear to follow at a high level and if you then pointed out the various address computations in the substeps it ought to be pretty straightforward.

A lot of my remaining questions concern how to support scrolling at least kinda correctly without basing everything off internal registers and how to render things tile by tile instead of pixel by pixel. But I may abandon that because it was mostly to avoid repeated fetches of the same data and I recently made a RENDER-CONTEXT object that can help with that. Maybe down the road at some point I'll make a cycle-accurate PPU. :)

Test ROMs

There aren't really good test ROMs because the ROMs that exist mostly assume you have the basics working and are testing tricky details. While I don't think a test ROM could be written since a lot of what needs testing are internal details of the PPU that weren't exposed to NES programmers, I do think a ROM coupled with some JSON dumps of what internal data should be visible after rendering for a frame or two would be incredibly useful. Because at some point I spent 3 days on a single bug because I wasn't incrementing a counter appropriately.

I'd like to think on this some more but I have some basic ideas. A lot of the difficulty is that in the 90s there were working emulators that did more abstract high level emulation both because PCs were less powerful and because less was known about the underlying hardware. That required workarounds of various sorts for accuracy and so they're frowned upon now. But as a result, I haven't found much high-level documentation of the rendering algorithm in the PPU. Everyone seems to point back to a frame timing diagram on the nesdev wiki. Which is great but I was hoping to write a 2500 lines of code readable NES implementation that doesn't require a solid understanding of latches and assembly to get a basic idea of how the thing worked.

But I believe the address calculations, among other things, can be expressed clearly (and tested!) in terms of the X,Y coordinates to be rendered as opposed to internal registers. More soon...

Hard to Follow Implementations

I'm still unsatisfied with my PPU implementation but it also isn't completely finished. I hope to have more to show here after scrolling is working and some refactoring is done.

A Word On Performance

The output resolution of the Nintendo was 256x240. At a high level, all the PPU is doing is looping from left to right (0-255), top to bottom (0-240) and deciding on a color for the current pixel, then outputting it once per frame. Of course, it has to do that 60 times a second and 256 * 240 * 60 is 3.6 million so pixel rendering needs to be pretty fast. I didn't have to do any optimizing to hit 60 frames per second but I was careful to write code that didn't allocate as I went and we're still using 50% CPU which is definitely more than I'd like.

Til Next Time

Wish me luck, lispers. Cheers. <3

Going Faster with Lisp

posted on 2017-09-17 16:10:00

For the first time in 3+ years, I'm working in earnest on a hobby project.

It feels like coming home, to be writing lisp and blogging again. Once again I'm playing with Nintendo emulation, don't ask why it's captured my imagination so. I'd like to briefly discuss what's brought me back and then we'll talk about what I learned writing lisp today.

My Absence

I haven't really worked on hobby projects since mid 2014. Even then my output was reduced substantially from 2012 when I lived alone and cl-6502/coleslaw had my full attention. I never stopped wanting to learn more, or loving lisp specifically, I just lost the energy to pursue it. That was due to (in rough order): Work, burnout, my relationship, and buying a house. Where burnout == a curiously strong blend of exhaustion, impostor syndrome, and unclear goals. It was scary at times when I wondered if I had lost my passion or commitment but ultimately good for me.

A lot of why I stalled out had to do with my old Nintendo emulator. I had made some bad assumptions, especially about how memory worked, due to not knowing much about systems programming or hardware when I started and didn't want to throw away everything I had to start fresh. cl-6502 had also felt very public so when progress had stalled before even being able to play a game that was quite embarrassing. I also didn't really know about test ROMs until way too late in the going.

But time heals all wounds and I have plenty of ideas. So here we are.

Design Changes

With cl-6502, I just focused on the CPU since that was something I had an inkling of how to approach. My biggest mistake was treating RAM as a 64k element array. The actual Nintendo used Memory Mapped I/O to talk to the graphics and sound cards. The only way to support that in famiclom was to overwrite the routines that read and wrote to RAM in cl-6502. It was unacceptable to me from both a design and performance perspective.

This time around, I'm using a separate object to represent the Memory Map so that when an CPU instruction reads or writes to an address, it'll actually get handled by the right part of the system: the RAM, Video Card, Sound, or cartridge. I'm also going to be focused on using test ROMs through as much of the process as I can. I'll write more about that in a future article but, long story short, TDD is hard to do when writing an emulator.

Lisp, Fast, OO: Pick 3

I managed to get cl-6502 running fast enough last time around but it was still 100x slower than Ian Piumarta's lib6502 written in C. There's no reason that has to be the case, I simply didn't know how to approach optimizing Lisp. I would use SBCL's statistical profiler, sprinkle compiler declarations in my code, re-profile, and pray. Today I'd like to focus on a few tricks for figuring out if declarations are helping or not and getting friendly with your disassembler. I'll also talk a little about why I wound up going with DEFSTRUCT over DEFCLASS.

lol, dis

Profilers are great for helping you figure out what parts of your code you spend the most time in. Once you've identified a function that needs to go fast, the next step is usually to add an optimize declaration. Something like:

(declare (optimize (speed 3) (safety 1))) ; or even (safety 0)

Recompiling the function afterward will result in the compiler printing out notes about what tripped it up while compiling the code. One thing I didn't realize back when I was working on cl-6502 (but seems obvious in retrospect) is that you can include optimize and type declarations in methods too! That said, it can be a pain to constantly write out different optimize and type declarations, recompile, and call disassemble on the code to see differences in the output. Additionally, there is not a portable way to disassemble methods, only their generic functions which is really just the dispatch machinery and not the work that you're interested in.

While Doug Hoyte's book Let Over Lambda is a bit controversial among lispers, he offers some good advice and good tools for remedying these points in Chapter 7. In particular, he supplies a read macro to quickly enable maximum optimization in a method or function and a regular macro to allow testing out type declarations effect on an anonymous function quickly at the REPL. I've taken the liberty of adding both to my .sbclrc file so I have easy access to them when I'm trying things out.

(defun enable-sharpf-read-macro ()
  (set-dispatch-macro-character #\# #\f
    (lambda (stream sub-char numarg)
      (declare (ignore stream sub-char))
      (setf numarg (or numarg 3))
      (unless (<= numarg 3)
        (error "Invalid value for optimize declaration: ~a" numarg))
      `(declare (optimize (speed ,numarg)
                          (safety ,(- 3 numarg)))))))

(defmacro dis (args &rest body)
  (flet ((arg-name (arg)
           (if (consp arg)
               (cadr arg)
               arg))
         (arg-decl (arg)
           (if (consp arg)
               `(type ,(car arg) ,(cadr arg))
               nil)))
    (let ((arglist (mapcar #'arg-name args))
          (declarations (mapcar #'arg-decl args)))
      `(disassemble
         (lambda ,arglist
           (declare ,@(remove nil declarations))
           ,@body)))))

I also dug around to see if there was a way to get disassembly for a single method and found a helpful thread on Google Groups from which I built a little function for disassembling the "fast-function" commonly invoked for a method.

(defun disasm-method (name specializers)
  "E.g. (disasm-method 'package:generic-fun '(class t))"
  (let* ((method (find-method name nil specializers))
         (function (sb-mop:method-function method))
         (fast-function (sb-pcl::%method-function-fast-function function)))
    (disassemble fast-function)))

A "Practical" Example

All code for this section is on the ground-floor branch on Github

Today I was working on memory mappers / cartridges for the NES emulator. Let's look at how I used these tools to optimize a method on the simplest mapper NROM. (Used in titles like Donkey Kong and the original Super Mario Brothers.) The method we'll be looking at is called load-prg. Put simply, it takes an address and loads a byte from the PRG section of the game cartridge.

Since any game will load data from the cartridge a lot we really want this to be a fast operation. And since it's loading from a static array, we would hope we can get this down to a handful of assembly instructions. Here's my initial implementation:

(defmethod load-prg ((mapper nrom) address)
  (let ((rom (mapper-rom mapper)))
    (if (= 1 (rom-prg-count rom))
        (aref (rom-prg rom) (logand address #x3fff))
        (aref (rom-prg rom) (logand address #x7fff)))))

You can see it takes an NROM mapper and an address and, based on the number of PRG banks in the cartridge, does a little math on the address and accesses the PRG with AREF. Let your eyes skim over the unoptimized disassembly:

CL-USER> (disasm-method #'clones.mappers::load-prg '(clones.mappers::nrom t))
; disassembly for (SB-PCL::FAST-METHOD CLONES.MAPPERS:LOAD-PRG
                   (CLONES.MAPPERS::NROM T))
; Size: 280 bytes. Origin: #x2290E8F5
; 8F5:       498B4C2460       MOV RCX, [R12+96]               ; no-arg-parsing entry point
                                                              ; thread.binding-stack-pointer
; 8FA:       48894DF8         MOV [RBP-8], RCX
; 8FE:       498B5805         MOV RBX, [R8+5]
; 902:       48895DE0         MOV [RBP-32], RBX
; 906:       8D43FD           LEA EAX, [RBX-3]
; 909:       A80F             TEST AL, 15
; 90B:       0F85F3000000     JNE L11
; 911:       8B4B01           MOV ECX, [RBX+1]
; 914:       4881F903FD5020   CMP RCX, #x2050FD03             ; #<SB-KERNEL:LAYOUT for CLONES.ROM:ROM {2050FD03}>
; 91B:       0F85C8000000     JNE L10
; 921: L0:   488B531D         MOV RDX, [RBX+29]
; 925:       BF02000000       MOV EDI, 2
; 92A:       E8411C1FFF       CALL #x21B00570                 ; GENERIC-=
; 92F:       488B5DE0         MOV RBX, [RBP-32]
; 933:       488B75E8         MOV RSI, [RBP-24]
; 937:       4C8B45F0         MOV R8, [RBP-16]
; 93B:       7456             JEQ L5
; 93D:       488B4B0D         MOV RCX, [RBX+13]
; 941:       8D46F1           LEA EAX, [RSI-15]
; 944:       A801             TEST AL, 1
; 946:       750A             JNE L1
; 948:       A80F             TEST AL, 15
; 94A:       7542             JNE L4
; 94C:       807EF111         CMP BYTE PTR [RSI-15], 17
; 950:       753C             JNE L4
; 952: L1:   488BFE           MOV RDI, RSI
; 955:       40F6C701         TEST DIL, 1
; 959:       7407             JEQ L2
; 95B:       488B7FF9         MOV RDI, [RDI-7]
; 95F:       48D1E7           SHL RDI, 1
; 962: L2:   4881E7FEFF0000   AND RDI, 65534
; 969:       8D41F1           LEA EAX, [RCX-15]
; 96C:       A80F             TEST AL, 15
; 96E:       7519             JNE L3
; 970:       8B41F1           MOV EAX, [RCX-15]
; 973:       2C85             SUB AL, -123
; 975:       3C74             CMP AL, 116
; 977:       7710             JNBE L3
; 979:       488BD1           MOV RDX, RCX
; 97C:       B904000000       MOV ECX, 4
; 981:       FF7508           PUSH QWORD PTR [RBP+8]
; 984:       E9AFEDA1FD       JMP #x2032D738                  ; #<FDEFN SB-KERNEL:HAIRY-DATA-VECTOR-REF/CHECK-BOUNDS>
; 989: L3:   0F0B0A           BREAK 10                        ; error trap
; 98C:       36               BYTE #X36                       ; OBJECT-NOT-VECTOR-ERROR
; 98D:       08               BYTE #X08                       ; RCX
; 98E: L4:   0F0B0A           BREAK 10                        ; error trap
; 991:       41               BYTE #X41                       ; OBJECT-NOT-INTEGER-ERROR
; 992:       30               BYTE #X30                       ; RSI
; 993: L5:   488B4B0D         MOV RCX, [RBX+13]
; 997:       8D46F1           LEA EAX, [RSI-15]
; 99A:       A801             TEST AL, 1
; 99C:       750A             JNE L6
; 99E:       A80F             TEST AL, 15
; 9A0:       7542             JNE L9
; 9A2:       807EF111         CMP BYTE PTR [RSI-15], 17
; 9A6:       753C             JNE L9
; 9A8: L6:   488BFE           MOV RDI, RSI
; 9AB:       40F6C701         TEST DIL, 1
; 9AF:       7407             JEQ L7
; 9B1:       488B7FF9         MOV RDI, [RDI-7]
; 9B5:       48D1E7           SHL RDI, 1
; 9B8: L7:   4881E7FE7F0000   AND RDI, 32766
; 9BF:       8D41F1           LEA EAX, [RCX-15]
; 9C2:       A80F             TEST AL, 15
; 9C4:       7519             JNE L8
; 9C6:       8B41F1           MOV EAX, [RCX-15]
; 9C9:       2C85             SUB AL, -123
; 9CB:       3C74             CMP AL, 116
; 9CD:       7710             JNBE L8
; 9CF:       488BD1           MOV RDX, RCX
; 9D2:       B904000000       MOV ECX, 4
; 9D7:       FF7508           PUSH QWORD PTR [RBP+8]
; 9DA:       E959EDA1FD       JMP #x2032D738                  ; #<FDEFN SB-KERNEL:HAIRY-DATA-VECTOR-REF/CHECK-BOUNDS>
; 9DF: L8:   0F0B0A           BREAK 10                        ; error trap
; 9E2:       36               BYTE #X36                       ; OBJECT-NOT-VECTOR-ERROR
; 9E3:       08               BYTE #X08                       ; RCX
; 9E4: L9:   0F0B0A           BREAK 10                        ; error trap
; 9E7:       41               BYTE #X41                       ; OBJECT-NOT-INTEGER-ERROR
; 9E8:       30               BYTE #X30                       ; RSI
; 9E9: L10:  488B512D         MOV RDX, [RCX+45]
; 9ED:       4883FA04         CMP RDX, 4
; 9F1:       7E11             JLE L11
; 9F3:       488B4125         MOV RAX, [RCX+37]
; 9F7:       81781103FD5020   CMP DWORD PTR [RAX+17], #x2050FD03  ; #<SB-KERNEL:LAYOUT for CLONES.ROM:ROM {2050FD03}>
; 9FE:       0F841DFFFFFF     JEQ L0
; A04: L11:  0F0B0A           BREAK 10                        ; error trap
; A07:       0A               BYTE #X0A                       ; OBJECT-NOT-TYPE-ERROR
; A08:       18               BYTE #X18                       ; RBX
; A09:       23               BYTE #X23                       ; 'CLONES.ROM:ROM
; A0A:       0F0B10           BREAK 16                        ; Invalid argument count trap

WOOF! 280 bytes of assembly, including a full CALL to a generic equality test, and two JMP instructions to other functions. Even without knowing any assembly, this seems like an awful lot of junk just for a measly array lookup! I think one valuable insight I got from Chapter 7 of Let Over Lambda was to disregard what I thought I know or didn't about assembly and just use my damn eyes. Doesn't this seem like a silly amount of code? Let's crank the optimization up:

(defmethod load-prg ((mapper nrom) address)
  #f
  (let ((rom (mapper-rom mapper)))
    (if (= 1 (rom-prg-count rom))
        (aref (rom-prg rom) (logand address #x3fff))
        (aref (rom-prg rom) (logand address #x7fff)))))

As soon as I recompiled this code, I got 6 notes from the compiler stating that it wasn't confident about the return value of (rom-prg-count rom) hence the generic equality test. It also wasn't confident what kind of array (rom-prg rom) was or if all the elements even shared a type! That will cause AREF to be slow. Even so, the generated assembly drops to 116 bytes since the #f read macro expands to a declaration with maximum speed (3) and minimum safety (0). It should go without saying that you only want to do this in code that A) really needs to be fast and for which, B) you're very confident about who will call it and how. Here's the disassembly:

CL-USER> (disasm-method #'clones.mappers::load-prg '(clones.mappers::nrom t))
; disassembly for (SB-PCL::FAST-METHOD CLONES.MAPPERS:LOAD-PRG
                   (CLONES.MAPPERS::NROM T))
; Size: 116 bytes. Origin: #x2290F6CB
; 6CB:       48895DF0         MOV [RBP-16], RBX               ; no-arg-parsing entry point
; 6CF:       488B4605         MOV RAX, [RSI+5]
; 6D3:       488945F8         MOV [RBP-8], RAX
; 6D7:       488B501D         MOV RDX, [RAX+29]
; 6DB:       BF02000000       MOV EDI, 2
; 6E0:       E88B0E1FFF       CALL #x21B00570                 ; GENERIC-=
; 6E5:       488B5DF0         MOV RBX, [RBP-16]
; 6E9:       488B45F8         MOV RAX, [RBP-8]
; 6ED:       7528             JNE L1
; 6EF:       488B500D         MOV RDX, [RAX+13]
; 6F3:       488BFB           MOV RDI, RBX
; 6F6:       40F6C701         TEST DIL, 1
; 6FA:       7407             JEQ L0
; 6FC:       488B7FF9         MOV RDI, [RDI-7]
; 700:       48D1E7           SHL RDI, 1
; 703: L0:   4881E7FE7F0000   AND RDI, 32766
; 70A:       B904000000       MOV ECX, 4
; 70F:       FF7508           PUSH QWORD PTR [RBP+8]
; 712:       E9E166A2FD       JMP #x20335DF8                  ; #<FDEFN SB-KERNEL:HAIRY-DATA-VECTOR-REF>
; 717: L1:   488B500D         MOV RDX, [RAX+13]
; 71B:       488BFB           MOV RDI, RBX
; 71E:       40F6C701         TEST DIL, 1
; 722:       7407             JEQ L2
; 724:       488B7FF9         MOV RDI, [RDI-7]
; 728:       48D1E7           SHL RDI, 1
; 72B: L2:   4881E7FEFF0000   AND RDI, 65534
; 732:       B904000000       MOV ECX, 4
; 737:       FF7508           PUSH QWORD PTR [RBP+8]
; 73A:       E9B966A2FD       JMP #x20335DF8                  ; #<FDEFN SB-KERNEL:HAIRY-DATA-VECTOR-REF>

A Fork in the Road

Those two JMP instructions and the generic equality CALL are still in the assembly though as you can see from the comments on the right hand side. Why? Because we didn't actually resolve any of the compiler's uncertainties about the code. We have to help it know what type of values it will be working with. The question is how to best do that. One way would be to just add a bunch of local type declarations in the method:

(defmethod load-prg ((mapper nrom) address)
  #f
  (let* ((rom (mapper-rom mapper))
         (prg (rom-prg rom))
         (prg-count (rom-prg-count rom)))
    (declare (type byte-vector prg)
             (type fixnum prg-count))
    (if (= 1 prg-count)
        (aref prg (logand address #x3fff))
        (aref prg (logand address #x7fff)))))

That will work and does generately substantially nicer code (82 bytes and no CALLs or JMPs). But boy, it forced us to completely restructure the method and, well, the new version feels a bit disjointed. The declarations stick out and distract from the underlying ideas. The alternative is to try and teach the compiler what types are returned by the accessor functions we're using to pull data out of the ROM. And this is where we come to the important difference about DEFCLASS and DEFSTRUCT from where I'm sitting as an emulator author.

Optimizing with Structs is Easier

(Ed. note 09/19/2017: Rainer Joswig left a very informative comment about Structs vs Classes and Optimizing with CLOS on reddit.)

Getting struct-related code to go fast is easier for a very specific reason. Both DEFCLASS and DEFSTRUCT allow you to optionally specify the types of their slots. Unfortunately, DEFCLASS does absolutely no optimization with this information, while DEFSTRUCT treats it as a guarantee and propagates it through the auto-generated slot accessors and, therefore, the rest of your code.

Now there's a good reason for this and I am certainly not advocating for using DEFSTRUCT by default. The reason is that DEFSTRUCT is not designed to be interactively redefined or changed at runtime unlike most of the language. DEFCLASS could have the types of its slots (or even the slots themselves) change at any time including runtime and so it isn't reasonable for it to treat the type declaration as a fact.

DEFSTRUCT has other downsides as well, including auto-generating a bunch of symbols in the current package among other things. It's clunkier to work with in several ways than DEFCLASS but for truly performance intensive stuff, the type declaration behavior makes it worth it from where I'm sitting. Just don't default to DEFSTRUCT in general. This message from the Rob Warnock Archive may also prove interesting.

This is something I always had questions about though and it was compounded a bit due to the fact that DEFSTRUCT is barely mentioned by Practical Common Lisp or Common Lisp Recipes. Practical Common Lisp is still the best way to learn the language in my opinion. I also honestly enjoy the things that are in the Common Lisp standard due to history but I'd never quite found an answer to "When should I use structs vs classes?" that I liked. Hopefully future lispers will be able to stumble on these notes (or parse the spec better than I did).

Modifying the ROM definition

Here's what our ROM struct looks like with the added type declarations:

(defstruct rom
  (pathname    nil :read-only t)
  (prg         #() :read-only t :type byte-vector)
  (chr         #() :read-only t :type byte-vector)
  (prg-count     0 :read-only t :type ub8)
  (chr-count     0 :read-only t :type ub8)
  (mirroring   nil :read-only t)
  (mapper-name nil :read-only t))

The previous version had no :type options and the default values were all nil. After changing the struct and recompiling, we can write the same version of load-prg as before but get much better generated assembly since the compiler knows the types returned by the struct accessors (and thus the array element type):

(defmethod load-prg ((mapper nrom) address)
  #f
  (let ((rom (mapper-rom mapper)))
    (if (= 1 (rom-prg-count rom))
        (aref (rom-prg rom) (logand address #x3fff))
        (aref (rom-prg rom) (logand address #x7fff)))))

; disassembly for (SB-PCL::FAST-METHOD CLONES.MAPPERS:LOAD-PRG (CLONES.MAPPERS::NROM T))
; Size: 90 bytes. Origin: #x22910BDE
; BDE:       488B4005         MOV RAX, [RAX+5]                ; no-arg-parsing entry point
; BE2:       488B501D         MOV RDX, [RAX+29]
; BE6:       4883FA02         CMP RDX, 2
; BEA:       7528             JNE L2
; BEC:       488B400D         MOV RAX, [RAX+13]
; BF0:       F6C101           TEST CL, 1
; BF3:       7407             JEQ L0
; BF5:       488B49F9         MOV RCX, [RCX-7]
; BF9:       48D1E1           SHL RCX, 1
; BFC: L0:   4881E1FE7F0000   AND RCX, 32766
; C03:       48D1F9           SAR RCX, 1
; C06:       0FB6540801       MOVZX EDX, BYTE PTR [RAX+RCX+1]
; C0B:       48D1E2           SHL RDX, 1
; C0E: L1:   488BE5           MOV RSP, RBP
; C11:       F8               CLC
; C12:       5D               POP RBP
; C13:       C3               RET
; C14: L2:   488B400D         MOV RAX, [RAX+13]
; C18:       F6C101           TEST CL, 1
; C1B:       7407             JEQ L3
; C1D:       488B49F9         MOV RCX, [RCX-7]
; C21:       48D1E1           SHL RCX, 1
; C24: L3:   4881E1FEFF0000   AND RCX, 65534
; C2B:       48D1F9           SAR RCX, 1
; C2E:       0FB6540801       MOVZX EDX, BYTE PTR [RAX+RCX+1]
; C33:       48D1E2           SHL RDX, 1
; C36:       EBD6             JMP L1

Finally, we can improve things just a bit by promising that the address we call the load-prg method with will be an unsigned 16-bit value since the 6502 only has a 64k address space:

(defmethod load-prg ((mapper nrom) address)
  #f
  (declare (type ub16 address))
  (let ((rom (mapper-rom mapper)))
    (if (= 1 (rom-prg-count rom))
        (aref (rom-prg rom) (logand address #x3fff))
        (aref (rom-prg rom) (logand address #x7fff)))))

; disassembly for (SB-PCL::FAST-METHOD CLONES.MAPPERS:LOAD-PRG (CLONES.MAPPERS::NROM T))
; Size: 66 bytes. Origin: #x22910DDE
; DDE:       488B4005         MOV RAX, [RAX+5]                ; no-arg-parsing entry point
; DE2:       488B501D         MOV RDX, [RAX+29]
; DE6:       4883FA02         CMP RDX, 2
; DEA:       751C             JNE L1
; DEC:       488B400D         MOV RAX, [RAX+13]
; DF0:       4881E1FE7F0000   AND RCX, 32766
; DF7:       48D1F9           SAR RCX, 1
; DFA:       0FB6540801       MOVZX EDX, BYTE PTR [RAX+RCX+1]
; DFF:       48D1E2           SHL RDX, 1
; E02: L0:   488BE5           MOV RSP, RBP
; E05:       F8               CLC
; E06:       5D               POP RBP
; E07:       C3               RET
; E08: L1:   488B400D         MOV RAX, [RAX+13]
; E0C:       4881E1FEFF0000   AND RCX, 65534
; E13:       48D1F9           SAR RCX, 1
; E16:       0FB6540801       MOVZX EDX, BYTE PTR [RAX+RCX+1]
; E1B:       48D1E2           SHL RDX, 1
; E1E:       EBE2             JMP L0

Updates

(Ed. note 09/19/2017: Some additional speedups have been made since this article was published.)

Paul Khuong was kind enough to note that SBCL was unable to hoist the (logand address xxx) computation out of the conditional. This duplication can be seen in the disassembly from the two MOV .. AND .. SAR .. MOVZX blocks. Doing so improved the assembly a bit further to 51 bytes. Reflecting on it further, I realized there's no need for a conditional expression at all!

In NROM cartridges, they can either have 1 or 2 PRG banks each of which are 16k. Because the 6502 has a 64k address space and the cartridge data begins at 32k, an NROM cartridge with only 1 PRG bank doesn't actually fill the address space. In our load-prg method, we just want to make sure that if we're given a higher address like 54321 that we wrap that around to not run off the end of our 16k worth of PRG. To do that, we can just logical AND the address with (1- (length array)).

Doing that eliminates the branch and results in a nice, lean 40 bytes for our final disassembly:

(defmethod load-prg ((mapper nrom) address)
  #f
  (declare (type ub16 address))
  (let* ((rom (mapper-rom mapper))
         (end-of-rom (1- (length (rom-prg rom))))
         (wrapped-address (logand address end-of-rom)))
    (aref (rom-prg rom) wrapped-address)))

; disassembly for (SB-PCL::FAST-METHOD CLONES.MAPPERS:LOAD-PRG (CLONES.MAPPERS::NROM T))
; Size: 40 bytes. Origin: #x22844CCE
; CE:       488B4005         MOV RAX, [RAX+5]                 ; no-arg-parsing entry point
; D2:       488B500D         MOV RDX, [RAX+13]
; D6:       488B52F9         MOV RDX, [RDX-7]
; DA:       4883EA02         SUB RDX, 2
; DE:       4821D1           AND RCX, RDX
; E1:       488B400D         MOV RAX, [RAX+13]
; E5:       48D1F9           SAR RCX, 1
; E8:       0FB6540801       MOVZX EDX, BYTE PTR [RAX+RCX+1]
; ED:       48D1E2           SHL RDX, 1
; F0:       488BE5           MOV RSP, RBP
; F3:       F8               CLC
; F4:       5D               POP RBP
; F5:       C3               RET

Wrap Up

There's a lot of work left to do on the (new) emulator but I'm writing code again, having fun, learning, and using lisp and that's the most important part to me. If you made it this far, thanks for reading. Let me know what you think and happy hacking!

Labor Day Link Fixup

posted on 2017-09-04 11:18:00

Ancient History

For whatever reason, yesterday seemed a good day to decommission Linode 18032 that I bought way back in February of 2009 and had been using to run redlinernotes.com ever since. I bought redlinernotes.com in June 2007 in the wake of my first serious breakup (with Sonya Z) but ran it out of my parents basement up until getting the linode.

Ever since, though my knowledge increased, I didn't bother revamping the linode other than one reinstall with Ubuntu 12.04 when I first migrated my blog away from Wordpress to Coleslaw. I've been gradually moving my online presence towards kingcons.io and making plans to get back to blogging in earnest the past few weeks. But even though kingcons.io was made with (semi-workable) ansible roles, redlinernotes.com was still a big hand rolled minefield with years of digital detritus to boot. Nothing like a 3 day weekend to clean out your digital woodshed.

A Fresh Start

After deleting a lot of crap, I moved the statically hosted documents over to kingcons.io, added an nginx vhost, swapped the DNS over, and nuked the old linode. There was one last thing to do though. When I moved over to coleslaw from Wordpress years ago, I didn't bother fixing up the old links. So I have a bunch of ".post" documents in my blog's git repo that reference expired wordpress links to other posts (like "blog/?p=1234") instead of the slug coleslaw would assign the post.

I guess I didn't care enough at the time or was too focused on coleslaw itself to worry about "legacy content". But I found a wordpress XML backup and figured I might as well fix up the dead links today while I was at it. All while rolling my eyes at dealing with XML.

Link Fixup

Since coleslaw is designed to be backed by git as a content store, I started by grepping through the blog posts to get a list of all the old wordpress post IDs I linked to.

grep -Eo "redlinernotes.com/blog/\?p=(\d+)" *.post | cut -d "?" -f 2

Armed with that, I could tackle digging into the wordpress XML backup to map the post IDs to coleslaw generated titles. Shinmera has been writing stupid amounts of good lisp code over the past few years including Plump, an XML parser. I never completely gelled with CXML back in the day so I figured I'd give plump a go. The following assumes you have a decent lisp installed (I heartily recommend SBCL) and quicklisp.

(ql:quickload 'coleslaw)
(ql:quickload 'plump)

;; Make sure to use a pathname (the #p) for this, plump will treat a plain string as XML to be parsed, not a file.
(defvar *wp-xml* #p"/Users/brit/projects/linode-retirement/wordpress.2010-09-14.xml")

(defvar *doc* (plump:parse *wp-xml*))

I was actually stumped here for a bit because I used a string instead of a pathname as the argument to PARSE and it took me a few minutes querying and getting no results before I looked in the slime inspector and realized the doc hadn't parsed as expected. Once I had this though, it was pretty straightforward to build a hash of IDs to titles...

(defvar *posts* (plump-dom:get-elements-by-tag-name *doc* "item"))

;; For those wondering, :test #'equal is there so we can use string keys. Read Practical Common Lisp or google to learn more.
(defvar *post-id-map* (make-hash-table :test #'equal))

(defun extract (key node)
  (let ((value (first (plump-dom:get-elements-by-tag-name node key))))
    (plump:text value)))

;; Yes, I'm using a private coleslaw function here but I wrote coleslaw so ... uh ... do what I want! 
;; And in case you were wondering, handler-case is lisp's try/catch equivalent and I'm pretty much doing "rescue nil" here.
(defun fixed-title (title)
  (handler-case (coleslaw::slugify title)
    (simple-error () :junk)))
               
(loop for post in *posts*
      do (let ((title (extract "title" post))
               (id (extract "wp:post_id" post)))
           (setf (gethash id *post-id-map*) (fixed-title title))))

And now we're good to update the existing posts to have proper relative links to the content we actually want.

(ql:quickload 'cl-ppcre)

;; Can you tell I did all this in one repl session and just copy pasted it into this blog post?
(coleslaw::load-config "/Users/brit/projects/improvedmeans/")
(coleslaw::load-content)

;; Cute sidenote, since ppcre turns this regex into to a tree of closures which SBCL compiles, this can be #'disassemble-d.
;; Also, it's pretty fast.
(defvar *wp-url-regex*
  (cl-ppcre:create-scanner "w{0,3}?\.?redlinernotes\.com/blog/\\?p=(\\d+)"))

(defstruct counts
  (invalid-slug   0 :type fixnum)
  (post-not-found 0 :type fixnum)
  (updated-url    0 :type fixnum))

(defvar *results* (make-counts))

(defun invalid-slug-p (slug)
  (or (null slug)
      (every #'digit-char-p slug)))

(defun mismatch-p (slug)
  (let ((key (make-pathname :directory '(:relative "posts")
                            :name slug :type "html")))
    (null (gethash key coleslaw::*site*))))

(defun slug-for-match (text start end match-start match-end reg-start reg-end)
  (declare (ignore start end))
  (let* ((id (subseq text (aref reg-start 0) (aref reg-end 0)))
         (match (subseq text match-start match-end))
         (slug (gethash id *post-id-map*))
         (new-url (concatenate 'string "blog.kingcons.io/posts/" slug ".html")))
    (cond ((invalid-slug-p slug)
           (incf (counts-invalid-slug *results*))
           (format t "Couldn't find valid slug for post id: ~d~%" id)
           "/&")
          ((mismatch-p slug)
           (incf (counts-post-not-found *results*))
           (format t "Not found in site content: ~A~%" slug)
           "/&")
          (t
           (incf (counts-updated-url *results*))
           (format t "Replacing ~A with ~A~%" match new-url)
           new-url))))

(coleslaw::do-files (path "/Users/brit/projects/improvedmeans/" "post")
  (let* ((text (alexandria:read-file-into-string path))
         (updated-post (cl-ppcre:regex-replace-all *wp-url-regex* text #'slug-for-match)))
    (with-open-file (out path :direction :output :if-exists :supersede)
      (format out "~A~%" updated-post))))

(format t "~%~%===RESULTS===~%")
(dolist (type '(invalid-slug post-not-found updated-url))
  (let ((symb (alexandria:symbolicate 'counts- type)))
    (format t "~A: ~D Posts~%" type (funcall symb *results*))))

And there you go. A few links are still broken but things are generally improved, I'm down to 1 linode instead of 2, and I had a bit of fun on a lazy Sunday.

===RESULTS===

INVALID-SLUG: 18 Posts

POST-NOT-FOUND: 7 Posts

UPDATED-URL: 58 Posts

In Memoriam

posted on 2017-07-26 07:36:00

I had a dream last night but don't remember much of it. I was the commander of a small starship I think. I don't know if we were civilian or military. I spent some time speaking to staff and checking various stations. And then the new recruits started arriving, nervous and eager. I think Owings was there welcoming them with me.

At some point, as the arrivals became distracted with idle chat, I flipped on the engines with a switch. They hummed to life and we all felt the ship rise slightly beneath us, rocking slightly, our hearts in our throats. A number of recruits shifted to regain their balance. The room fell quiet. I smiled and woke up.

The only part of the dream that matters is that brief moment of floating. Before the voyage really starts. Before we know each other.

I know it's silly but it can still be important.

I believe in what we did at The Iron Yard.

This blog covers 2015, Books, Butler, C, Dad, Discrete Math, Displays, Education, Erlang, Essay, Gaming, Gapingvoid, HTDP, Hardware, IP Law, LISP, Lecture, Lessig, Linkpost, Linux, Lists, MPAA, Milosz, Music, Neruda, Open Source, Operating Systems, Personal, Pics, Poetry, Programming, Programming Languages, Project Euler, Quotes, Reddit, SICP, Self-Learning, Uncategorized, Webcomic, XKCD, Xmas, \"Real World\", adulthood, apple, careers, choices, clones, coleslaw, consumption, creation, emulation, fqa, games, goals, haltandcatchfire, heroes, injustice, ironyard, linux, lisp, lists, math, melee, metapost, milosz, pandemic, personal, poetry, productivity, professional, programming, ragequit, recreation, reflection, research, rip, strangeloop, work

View content from 2023-07, 2023-02, 2022-12, 2022-06, 2022-04, 2022-03, 2022-01, 2021-12, 2021-08, 2021-03, 2020-04, 2020-02, 2020-01, 2018-08, 2018-07, 2017-09, 2017-07, 2015-09, 2015-05, 2015-03, 2015-02, 2015-01, 2014-11, 2014-09, 2014-07, 2014-05, 2014-01, 2013-10, 2013-09, 2013-07, 2013-06, 2013-05, 2013-04, 2013-03, 2013-01, 2012-12, 2012-10, 2012-09, 2012-08, 2012-06, 2012-05, 2012-04, 2012-03, 2012-01, 2011-10, 2011-09, 2011-08, 2011-07, 2011-06, 2011-05, 2011-04, 2011-02, 2011-01, 2010-11, 2010-10, 2010-09, 2010-08, 2010-07, 2010-05, 2010-04, 2010-03, 2010-02, 2010-01, 2009-12, 2009-11, 2009-10, 2009-09, 2009-08, 2009-07, 2009-06, 2009-05, 2009-04, 2009-03, 2009-02, 2009-01, 2008-12, 2008-11, 2008-10, 2008-09, 2008-08, 2008-07, 2008-06, 2008-05, 2008-04, 2008-03, 2008-02, 2008-01, 2007-12, 2007-11, 2007-10, 2007-09, 2007-08, 2007-07, 2007-06, 2007-05