MySQL to MongoDB Query Translator

A few weeks ago, the first-ever RJMetrics hackathon took place at our Philadelphia headquarters. I decided to throw my hat into the ring with a project I’d been thinking about for a while: a MySQL to MongoDB query translator.

This was a unique challenge because MongoDB and MySQL are very different technologies that store data in very different ways. To some, translating between them might seem like a non-sequitur. However, I knew there was a use case because of my personal experience learning MongoDB. I would often think about queries in terms of SQL syntax, and a translator like this would have greatly softened the learning curve.

The final product is available at our Query Mongo site, and I encourage you to give it a try. It’s not perfect, but we hope it will be a helpful learning tool for the many people who have SQL experience and are getting started with MongoDB.

In this blog post, I’ll provide some insights into how this tool works.

Continue reading

The Future of Web Dev is (almost) here!

 

Hoverboards probably won’t be on the shelves until 2015, or possibly 2014, but alpha versions of amazing, bleeding-edge web development frameworks are available today!

Recently, there have been some exciting developments in this space. Several teams are working on their own solutions with the goal of drastically simplifying the process of developing a web application.

Derby

Meteor

Firebase

SocketStream

Capsule & Thoonk

Space Magic

These frameworks boast a few common features I am very excited about:

Sharing code between client and server

I feel terrible every time I have to duplicate model validation logic in both a server-side language and javascript. These new frameworks are based on javascript, or languages that compile to javascript (like coffeescript), which means everything you write can be run on the server or in the browser.

Notably, clojure, with the invention of clojurescript, also has the capability to compile to javascript, although I am not aware of anyone working to leverage the concept of real-time model syncing and live binding into a framework. I would love to see this.

Automatic real-time model syncing and view rendering

Every web app I’ve written has had semi-specialized code dedicated to translating UI interactions into changes to a data model, and then communicating those changes to and from the server. These frameworks allow you to define UI-model bindings declaritively and then handle propagation between UI, model and server automatically and in real time. And, if one client makes a change to a model that is also powering another client’s UI, the second client receives the updated model data and has their UI re-rendered automatically as well. These are powerful ideas and will simplify or eliminate many of the typical complications that currently hinder web development.

What’s the catch?

The catch is that these technologies are still very alpha. They are either missing important features, very buggy, or regularly making breaking changes to their APIs. I’ve spent some time with Derby in particular, and it’s definitely bleeding-edge (because I got cut, bad). But, that hasn’t lessened my strong belief that the fantastic ideas implemented in these solutions will be part of mainstream web development in the near future. I’m rooting for all of these guys.

Parallel SSH and system monitoring in Clojure

During my 10% time, I created two simple clojure tools to aid in basic sysadmin tasks. Today I’m open sourcing them on github and clojars.

parallel-ssh

The first tool I built is a library for running commands in parallel on multiple servers. It takes a BASH command and a csv of server names to run the command on. Internally, one clojure agent is spawned per server and each agent is responsible for running the command and storing the result. After all the agents have completed, or a specified timeout is reached, the agents are dereferenced and their output is returned. Currently, I just shell out to run ssh and I make the assumption that password-less login is available.

This library also has a command line interface:

 

I found it useful to wrap that in a BASH script that would run a command on all of our servers. This is clearly not a replacement for sophisticated server management tools like puppet, but it is helpful when you quickly want to an answer a question such as: “How much disk space is free?” or “How many servers is this process running on?”, etc..

server-stats


The second tool is a micro-framework built on top of parallel-ssh. Similar to python’s fabric, it allows the user to define custom commands to be run on a specified group of servers. It also has the capability to respond to the results of the command run based on custom triggers. Here’s an example configuration file:

First we define the ssh username to be used and our server groupings:

 

Now we can start adding commands. Here we add a command called ‘top’ that will be run only on web-servers and app-servers:

 


Note that the doc string is used in the auto-generated usage page, so you should never have to open the config file to figure out what a command does. We can now run this command from the command line:
We can make things a little more interesting by adding alert triggers. First we need to define an alert handler function. An alert handler takes 3 arguments: the alert message, the name of the server, and the output from the command that was run. Here we add a handler called ‘email’ that will send us an email when a trigger condition is met:
Now lets define a command and trigger that will use this alert:
This command has an extra field called ‘alerts’; this is an array of trigger conditions for this command. The command ‘disk’ only has one trigger, which states “when the Use% column of ‘df -ah’ is greater than 85%, send an email with the message ‘Disk space over 85% full’”. Heres a breakdown of an alert:

  • ‘column’ is used for commands that return column-formatted output (eg. df, iostat, top), and it instructs server-stats to look at a specific column for the value. If it is not specified it will assume the command output is a scalar value.
  • ‘value-type’ tells server-stats how to parse the command result string in to a clojure value. Right now there are only three possible value types: percent, bool, and number.
  • ‘handlers’ is a vector of alert handlers to call when this condition is met. In this case, it is just the email handler.
  • ‘msg’ is the alert message that gets passed as the first argument to the handler function.
  • Finally, ‘trigger’ actually defines the condition that has to be met. It is a tuple which has a Boolean operator and a value to compare against.

Additionally, you can define a global function to be called whenever a command can not be successfully completed for some reason (eg., server timesout). Here we send a text message using Twilio whenever that happens:


 

 

conclusion

Currently this is just used for basic server monitoring, but this could easily be used for much more advanced reactive behavior. Building this in clojure was a lot of fun and pretty easy since clojure has macros, easy to use concurrency, higher order functions, and full access to java libraries. The one downside of using clojure on the command line is you have to eat the JVM startup time on every run.

resources

http://clojars.org/server-stats

http://clojars.org/parallel-ssh

https://github.com/RJMetrics/Parallel-SSH

https://github.com/RJMetrics/Server-Stats



Feedback loops, tight ones: interface design, business management & tickling

Here at RJMetrics, we take our clients’ data and present it back to them in a more readily understandable and actionable form. What happens after that is up to the customer, but in an ideal situation, they will be able to make decisions and adjust behavior based on what the data is telling them. This amounts to a feedback loop.

The simple diagram provided shows the cyclical relationship between making a decision, executing an action and observing the reaction from the environment. We come in at the last piece of the loop, where the reaction must be clearly perceived in order to make the most informed decision possible.

From the moment we begin to learn to control our own bodies, we automatically participate in a feedback loop where we attempt a movement, observe the result and learn to do better. They are so ubiquitous and so intuitive to us that it is almost too easy to overlook their importance. In the past few months, my attention has been drawn back to them by various unrelated articles and conversations.

I got (too) excited thinking all the ways that feedback loops exist at various levels of reality, from the concrete physical to the abstract. I reaffirmed my belief that they are one of the magic truths of the universe. Then I calmed down and realized I needed to focus and come up with a reasonably coherent blog post.

I hope you enjoy my brain’s dump on the matter.

 

Bret Victor – loop tightener and maker of awesomeness

Bret Victor understands feedback loops. He’s dedicated his life to them and their improvement. It’s a noble quest. Here’s a quote from a recent talk he gave in Montreal this year:

“Here’s something I’ve come to believe: creators need an immediate connection to what they are creating. That’s my principle. Creators need an immediate connection to what they create. And what I mean by that is, when you are making something, if you make a change or you make a decision, you need to see the effect of that immediately. There can’t be any delay and there can’t be anything hidden.”

 

Bret Victor – Inventing on Principle from CUSEC on Vimeo.

 

Bret is talking about tight feedback loops. The creative process is a feedback loop. The creator performs some action on an object and the object responds by taking on a new form. But, this isn’t good enough for Bret. For the creative process to be intuitive and efficient, the response must be perceived immediately and clearly – not tomorrow and not with anything missing or obscured.

The underlying message of Bret’s talk is that you have the option to choose a single principle to focus your life and career on. He spends the first half of the talk describing his own journey and where it has taken him, which includes several impressive software examples. The ones that stood out most to me were the developer tools.

Bret recognized that programming is a creative process and that it could benefit greatly from tighter feedback loops. He gives two examples, a game and a sorting algorithm, where he lays bare all relevant state data and a visual time line of how it is transformed by the code over time. His software allows you to make changes to the code and perceive how the state time line is affected in real time. If that sentence didn’t blow your mind, you should watch the video and try to get your mind blown that way (my favorite part starts around 11:30). His examples are extremely impressive and it is hard for me to imagine a future where these ideas have not had an impact on how code is written or how computer science is taught.

(If you’re interested, this guy implemented a version of Bret’s game with real-time code evaluation. It’s really cool. https://github.com/ibdknox/live-cljs) (Brett also has a paper I really like here http://worrydream.com/MagicInk/ )

This type of immediacy and transparency is generally taken for granted when talking about the physical world. Our ancestors evolved in an environment where they could strike one rock against another and see progress toward the shape of a blade or arrow point. They could use that arrow point to make a spear, poke it at a T Rex and get immediate feedback. This is the experience Bret wants for all our creative endeavors.

However, in more abstract areas, feedback loops are often slow or provide only partial information. In software, long compile times, or the steps involved in recreating a bug, limit the rate at which software can be improved or fixed. In business, it is not uncommon for there to be a long period between the design of a product and the first chance to observe real market reaction. And simply observing whether a product succeeded or flopped is insufficient: what about the hidden details? Why was the reaction positive or negative? And how can it be improved next time? While a creator is waiting for this information, if he is making decisions, they are uninformed at best.

Lean Startup – throwing my hype on the pile

Eric Ries is another champion of the tight feedback loop. In Lean Startup, he practically begs the startup community to avoid “achieving failure” by paying attention to feedback loops and not building a product that nobody wants. One of the core ideas of the book is that a business should iterate on a “build, measure, learn” loop and that it should do so as quickly and cheaply as possible.

To summarize the worst possible scenario:

- build a new product over the course of months (or years!)

- after releasing it, measure that there is no demand, profit or money left in the bank

- learn that your business has failed

The feedback loop above is as loose as it gets, you only go through it once and at the end you’ve achieved little or nothing. Instead, keep it tight: build the simplest thing possible, fail fast, learn fast.

Both Bret and Eric ask us to focus not just on our decisions and actions, but the context in which we make them; that we reject poor tools and methods and replace them with ones that respond as quickly and clearly as possible. My big point is, when you are creating, whether is be software, a new business or a melody on the piano, taking small, informed steps gets you to the goal faster.

 

Let’s get real

I wouldn’t feel satisfied with this post unless I dug a little deeper here and threw some cognitive science at you. Here you go.

Our brain is constantly using sensory feedback to improve our internal models of the world. For example, whenever our brain sends a motor signal (efference) to our body, to initiate some physical action, it also produces a copy, known as an efference copy.

http://en.wikipedia.org/wiki/Efference_copy

One purpose of the efference copy is to enable the brain to produce a forward internal model of what sensory feedback we should expect when performing some action. This estimate is compared to the actual feedback and any difference is attributed to external stimuli. This is sensory discrepancy in the chart above.

We use this sensory discrepancy to distinguish ourselves and our own actions from the rest of our environment. We have been refining this forward modeling process all our lives. You can imagine that at a very young age, our ability to make these estimations was crappy. This likely made it impossible to understand what effect our actions had. This would have been confusing and disorienting, but I suspect might be similar to what one feels when working with unfamiliar (and poorly designed?) software or in some other foreign environment.

Contrast this with how a master pianist must feel while playing. After years of practice and consistent feedback from the instrument, the response of piano must be part of his internal forward model. The piano feels like an extension of himself and any variation from expected performance will immediately call his attention. We feel the same way when carrying out daily tasks such as writing or anything that is just like riding a bike. The process feels completely natural.

I believe this sensation is the end goal for Bret Victor. That all creative tasks should feel this natural. But, when do we go wrong? In the case of efference copy, it happens in a tenth of a second. How do we know? Tickling.

 

When is touching myself not touching myself

Starting around the 11 minute mark of this TED talk, Daniel Wolpert explains his experiments with people tickling themselves. Typically, people are not very good at tickling themselves. It becomes much easier though if you have a robot helping who will mimic your actions but introduce some amount of delay. Wolpert found that with zero delay, participants reported an equal amount tickling sensation as they feel when tickling themselves directly. But, by introducing a delay as small as a tenth of a second, the reported tickling increased. By, three tenths of a second, the tickling was full on. The loop was loose.

To me, this indicates that our sense of having full control of our environment is very finely timed. If results deviate from expectations by tenths of a second, we begin to feel that alien forces are at work. This is consistent with the frustration (or even fear) that people often feel towards software they have to use. Designers should take this to heart.

Business owners should take it to heart too:

In A/B tests, [Amazon] tried delaying the page in increments of 100 milliseconds and found that even very small delays would result in substantial and costly drops in revenue.

http://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html

 

Bring it on home

I like feedback loops and I want RJMetrics to be part of yours. I’ve been part of a team here that has been focused on improving the user experience around creating and modifying your charts. During all the refactoring and redesigning, the importance of a responsive feedback loop in reaching this goal has been at the front of my mind. There are many changes coming, but I believe the most important one will be the new WYSIWYG component. We are building an interface where every change you make will be quickly reflected in the chart visualization you are working with. We are hoping to make the process of exploring and perceiving your data as intuitive as possible.

Challenging Dogmatic Development

Developers are an egotistical bunch, and as such we tend to deal in fantastical absolutes. These range from language choice to design patterns to naming conventions, and it seems like everybody has their opinion. Fortunately, development is a team sport. One developer’s personal preferences probably won’t sync up with his team’s best practices. So, in the best interest of teams everywhere, I want to enumerate some awful (and sometimes comical) dogmatic beliefs that make development downright painful.

Horrible Dev Dogma #1: Language x is inferior

C++ and PHP (among others) seem to suffer scorn in many developer circles. For example, in Peter Seibel’s Coders At Work, development gurus such as Jamie Zawinski and Brendan Eich describe C++ as being somewhere between the awful “just an abomination” and the slightly more generous “stupid”. So, some of the most famous and most talented developers in the world denounce C++ as useless, yet C++ is one of the most popular programming languages on the market, especially for Windows applications. In fact, Google’s codebase is made up almost entirely of C++. Does this mean Google is an abomination?

Part of the sentiment stems from the language, and part from developers who use it. C++ is oversaturated with shakily implemented features that empower bad developers to do things the wrong way. Thus we have Linus Torvalds ranting about C++ as the wrong language for git and mocking devs who would suggest its use in the process. Many interpret this as an attack directly on C++, but it is really an attack on bad developers (or, more exactly, developers who think git should be written in C++). These gurus are so sick of seeing bad devs write bad solutions in a mediocre language that they finally denounce that language altogether.

Bad developers are language agnostic, as are good ones. Let’s stop denouncing languages just because bad devs use them incorrectly.

Horrible Dev Dogma #2: I always use x formatting/naming scheme and so should you

Refactoring is an important part of maintaining a healthy codebase, but personal conventions are not applicable to a team project. Is it really necessary to name every variable after a Disney character? Do you have to use spaces when everyone else on your team uses tabs? There is no good reason for this kind of stubbornness. Learn what your team does, and do the same.

Horrible Dev Dogma #3: Everyone should use TDD/CDD/x

This is pure insanity. Not everyone should use testing driven development. I repeat: testing driven development is not for everyone. Nor is comment-driven, agile, XP or any other software development method. There is no perfect development method, and all of these have their pros and cons.

Software development methods should make the developer’s life easier, not harder. Additionally, the development method should bow to the developer, not the other way around. TDD is painful for some and convenient for others.

Pete Sergeant at Write More Tests has an interesting take on TDD which I find particularly appealing. TDD answers some interesting questions about software development, and most importantly it makes development easier in some cases and for some developers. By no means is it the ultimate solution to all development problems.


The keys to a successful dev team are flexibility and consistency. There is no perfect end-all, be-all solution to software development, so let’s stop pretending that our own personal preferences will fit everybody else and start challenging dogmatic development.

Our Living Style Guide (Writing maintainable HTML/CSS)

There’s a lot of talk about scaling apps for growth. Scaling server architecture, writing code that will handle millions of requests etc. but I rarely hear about scaling HTML/CSS. By this I don’t mean performance, I mean maintainability. Let’s take a look at this fact: Facebook currently has 6498 color declarations! You don’t get there unless there wasn’t a plan for maintainable markup from the beginning. Since that sounds like a big ol’ mess, I’m dedicated to avoiding that problem at RJ.

The solution, quite simply, is to start forming a set of rules, nomenclature and workflow to form an architecture.

The Goals:

  • Create a worflow for creating and understanding new markup.
  • Give developers who aren’t experts in CSS, a logical and consistent way to markup HTML without having to ask a designer.
  • Impose markup standards within a growing organization.
  • Make it obvious when something strays from being consistent with an organization’s style, and publicly shame us into correcting it. It’s an experiment, and hopefully generates feedback.
  • Save time, and thus, save $ for beer.

The Strategy:

1. Break UI components into objects conceptually. (Buttons, notices, pickers, etc)

2. Address those components in a unified stylesheet: common.scss (We use Sass with Compass, and it’s pretty awesome).

3. Drop common.css into the app and play whack a mole with the style inconsistencies that crop up.

This part ended up being WAY easier than I anticipated. The reason being that there wasn’t much semantic markup. Nearly everything was a div or span with a class on it. I love semantic markup, but this was a case where a lack of it had an unintended positive outcome. This alone deserves its own post. The biggest issue actually ended up being typography since there were so many declarations. We’re still sorting this out.

4. Isolate more objects and add them to common.css until all commonly used objects reside in one place. I like to think of common.css eating the other style sheets.

5. Create a living style guide that outlines what each object looks like, its class structure, and its markup. It’s using the same common.css that’s deployed in our app, so if it works in the style guide, it will work in the app.

Take adding a notice for example. All the developer needs to do is go to the style guide, copy the HTML, add the classes they need, and paste into the app. They can even add classes that don’t exist yet, and have someone like me go back and add the CSS later.

scalable app design

The style guide was inspired by and is based on Kyle Neath’s KSS and more recently takes a play from Pea.rs by SimpleBits. Nomenclature and organization are inspired by Nicole Sullivan’s OOCSS and Jonathan Snook’s SMACSS.

Check it out, and let us know what you think.

Update: If you’ve got an opinion on this, maybe you’d like to work with us! Apply here.

Lessons learned in error tracking

Here at RJMetrics we pride ourselves on providing one of the leading hosted solutions in business analytics. One of our product’s highlights is the ability to replicate and warehouse client’s data in our data servers. Replication is not an easy task. It is driven by a set of very complex algorithms spanning across a number of different modules. Such algorithms are prone to latent bugs and vulnerabilities. They are hidden away in edge cases, just waiting for the right time to reveal themselves.

Why logging is not enough

No matter how good of a developer you are, bugs are inevitable and the best way to deal with them is to always expect them. That is why every good software engineer should always provide some means of error tracking. Traditionally this is achieved through logging, also known as tracing. Logging has the benefit of allowing the development team to go back at any given time and trace back the exact code path an operation took.

Logging is not a silver bullet though, and for it to be useful an alert system has to be in place to make it obvious to the development team that something is wrong and someone has to attend to it. Without a good reporting mechanism, bugs in fundamental areas of the system are never discovered and resolved, and any further enhancements to it are made on the false assumption that the core of the system is reliable.

The problem with email alerts

Usually development teams handle error reporting by injecting email alerts whenever an exception is caught. These emails often go to a single recipient, or to a mailing list. The problem with email alerts going to a single person is simply NO transparency. The rest of the team will have no clue about these issues unless they are forwarded/delegated to the rest of the team.

Email alerts are not any more useful if they are sent to a set of developers because it provides no accountability. Everyone in that mailing list will just expect everyone else to attend to them. Furthermore such email alerts will eventually get lost in someone’s inbox with no good way of going back to them or tracking them.

Our take on error reporting

The same exact scenario was playing when I first started working here. One of the first assignments I was tasked with was to replace all of the annoying, at best, emails with a much better system that would allow us to be more aggressive and proactive in resolving bugs. The initial solution was:

1. Provide a web interface through which at any time we could see any errors that occurred in the replication process

2. The interface would include detailed information along with a full stack trace, exception messages, as well as the frequency at which an issue has been happening

3. Reporting would occur for every distinct combination of replication job type and construct.

You might rightfully say, “how was that a better solution than emailing everyone in the team!”. It wasn’t much different, but it allowed us to make our error reporting even more useful by just taking small incremental steps with no significant effort from our development team.

Our next step was to dedicate one person in every sprint cycle to attend to all the issues being reported, and to make sure that any false positives are filtered out and any critical issues are resolved. This was obviously already working much better that the email alert system but it was still not providing enough transparency to the rest of the team. Unless we all took the time to check, we wouldn’t know how many errors got generated and how many of them got resolved.

The solution was to integrate our error tracking and reporting mechanism with fogbugz. New tickets get created when replication job errors occur, and they are handled just like any other high priority tickets in our sprint. We can now track replication job errors through our familiar ticket tracking system and even get to use our own product to make a more intelligent analysis of bugs in our system.

Conclusion

In just a few sprint cycles we were able to bring down the number of replication job errors from affecting around 20 clients close to none at any given time. We were able to uncover and resolve some really obscure bugs and we can now be confident that when something goes wrong we will be addressing it at a timely manner and before it affects any of our clients.

The Human Element in Automated Software Testing

As with any startup, the excitement of improving and adding to our current product often overshadows more mundane aspects of software development, such as automated software testing. In response to this, here at RJMetrics we have recently been examining our current suite of automated tests and reevaluating our strategy towards testing. There are already lots of great blog posts about unit testing philosophy and best practices, so instead I’d like to share some of our personal experiences navigating the human elements of automated software testing.
  • Testing can be a divisive issue. Each developer subscribes to his or her own philosophy of software development, and this includes testing. You will never get everyone to agree on these issues and could spend hours debating the goals and merits of testing, so my best advice is to decide upon and document a team philosophy early on. This includes how your team defines the basic terms such as unit and integration test, and classifying the types of tests that are important for your product. You should also clarify the main goals of your tests (to catch bugs, aid in refactoring, etc), as this affects how tests are written and what is tested. Once this is in place and everyone is on the same page, you can move forward with more meaningful discussions.
  • A smaller number of thoughtful unit tests is infinitely better than a large number of poorly written unit tests. In addition to failing to catch real bugs, bad unit tests break as a result of unrelated code changes. Fixing such tests slows down product development by taking time away from other projects, and it takes an even worse psychological toll on the team. Spending minutes or hours debugging a failing unit test is enough to turn the most avid supporter sour on testing. To avoid such tests, everyone should be familiar with unit testing “best practices” and any testing pitfalls specific to your codebase should be addressed as early on as possible.
  • People are more likely to write and run unit tests if its easy to do. Try to simplify the process as much as possible by providing a collection of helper functions and objects that will allow developers to focus on testing the code at hand. For us, this means providing a set of “stock” objects, ready to be used in a test.
Bearing in mind the experiences above, we are moving forward to improve and simplify our automated testing. There is still much more for us to learn and we will periodically reevaluate and keep everyone posted on the success of our testing initiative in catching bugs, refactoring old code and aiding in new code development.

PHPUnit Tips: Stubs vs Mocks

I recently spent some time digging into the PHPUnit mocking framework to clarify my own understanding and better decide how we can improve our unit testing tools. I’ve found the PHPUnit documentation on the subject to be inadequate for anything but the most basic use cases. (The full API documentation does not include the mocking framework, as far as I can tell.) I’ll try to clarify the one aspect of this that confused me the most and caused me to lose the most time.

Stubs vs. Mocks

This is touched on in the documentation, but I’m not a fan of the way it’s described. “Mocks” are objects which have some of their methods replaced by stubs (methods with no behavior). “Stubs” are objects which have stubs for all methods, with expectations optionally placed on certain methods. As far as I can tell, the distinction between a mock and a stub comes when you provide a list of explicit methods to stub, or not. When you provide a list of methods like so:

$mock will defer to the original class’s methods for all method calls except for “methodOne” and “methodTwo”. These methods will always result in successful calls, but will not return any values.

When you don’t provide any methods, you get a stub with no functionality. This object has the ability to act in place of the mocked class wherever it is used, but has no functionality on its own. Once you create this object, you can add expectations and return values to force it to behave in ways that are useful to you:

This strategy allows you to build piecemeal mocks that build functionality you need as you go. I find this the most useful way to use PHP object mocking. If you’re creating objects which need to be “partially mocked” in order to test, you might want to reevaluate the structure of your objects–possibly moving those partially mocked methods into another component, which can then be fully stubbed.

At RJMetrics, I’ve been experimenting with a mock dependency container which will provide stubs for all our objects. This will give us the greatest flexibility in terms of being able to add units of functionality needed for a specific test, while still maintaining internal consistency with all of our dependent objects.

Lexical vs Dynamic Scope in Clojure

Here at RJMetrics we recently added 10% time to our development cycle. I decided to do my project in clojure because I needed easy concurrency and I wanted to learn a new skill. Since I have never worked in a lisp-based language before, I found the fact that clojure has two ways to resolve a symbol’s scope to be a little confusing at first. I found many posts that helped explain the difference, but few took a beginner approach. I hope this post will clear up any confusion that new clojurians may have.

Background

A “scope” in computer science is a context that is created to encapsulate the values of variables (or symbols in clojure) to smaller pieces of code. Without any scopes, every variable would be global. Can you imagine if every time you used the variable x or i you had to worry about interfering with another piece of code? Luckily, most sane languages provide a means of creating and restricting the scope of variables. Some languages (eg. Clojure, Common Lisp, Perl) have two means of controlling scope, lexically (aka static) and dynamically.

Clojure Terminology

These are a few basic clojure terms you will need to know to understand the rest of this post.

  • A symbol is similar to an identifier in other languages. Symbols are used to associate values with common names. In an imperative language like PHP you might say something like “$x = 5;”. Now the variable $x equals 5, in clojure semantics I would say the symbol x is associated with the value 5. Clojure has an opinionated philosophy on separating identities and values.
  • A value is just actual data, such as a number, string, function (code is data), vector, object, etc..
  • A var is a way of tying a symbol to a potenitally changing value; they are created using the def macro. This will be covered in more depth when I cover dynamic scope.

Lexical Scoping

The lexical scope of a symbol is the “textual region” (aka code block) where the symbol definition occurs, plus all child code blocks. You can imagine your code creating a tree of code blocks that the compiler can traverse. What defines a code block is dependent on the language. Lexical scoping is by far the most common amongst all modern ALGOL-influenced languages (C, Java, etc). When the compiler is trying to resolve a reference to a variable, it will first look in the inner-most scope and continue up the levels of scope (code blocks) until it finds its declaration. Lexical scope is sometimes called static scope since all symbol references can be resolved at compile-time, as opposed to dynamic scope which needs runtime information. In this C example below the variable i is available within the for and while loops, because the while loop is below the for loop. However, i is not available outside of the for loop since that is above the scope in which the variable was defined.

This is the scope you will be using in clojure the vast majority of the time. The main difference is the scope blocks are declared explicitly by the programmer in clojure rather than by the syntax of the language. In clojure, a lexical scope region is defined by a let block. In the code example below, x, y, and add are all symbols that are associated with the values 3, 5, and a function that adds all of its arguments, respectively (remember functions are first-class values).

The lexical environment is encapsulated by the parentheses (aka parens) surrounding the let block, which means as soon as I close the let statement, those symbols go back to their original values. Here’s a more complex example:

The def statement on the first line creates a new global var with the value 2 and associates it with the symbol z, which is similar to how constants would be used in other languages. In this example, we defined a new let block within the previous one. Remember, when the compiler tries to resolve a symbol it will recursively go up each scope block until it finds the first symbol definition that matches. In the inner let block we used the symbol z, which already had a global definition. In this case, z is being shadowed by the value 10. As soon as we exit the let block, the value of z will return to its global definition. So by using let blocks to declare your local symbols, you never have to worry about overwriting some global value like you would in PHP or JavaScript. One thing to remember about lexical scoping is that it only deals with the text inside of the let block. So even though I called add-3-to-z while the local value of z is 10, the text defining the function add-3-to-z is outside of the let block so it defers to the global value of z.

In this last example, while inside a let block, I create and return a function that takes 1 argument and adds it to the value of x. Since the code that created the function was within a let block, the value of x is 1 even if the function is called outside of the let. Due to the fact this function retains its scope at the time of creation, it is said to “close” over the lexical environment. This type of function can be referred to as a lexical closure, or just closure for short (this is where clojure gets its name).

Dynamic Scope

Dynamic scope is another method of resolving symbol names. A dynamic scope environment is created using the binding macro, which syntactically looks like let, but has some big differences:

  1. You can’t create new symbols. Every symbol has to be associated with a previously initialized var.
  2. The initialized vars also have to be defined as “dynamic”.
  3. The new bindings affect the entire call stack, not just the immediate text region.
  4. The bindings that are created are pushed onto a thread-local stack and cannot be shared between multiple threads.

We can’t make up new symbols, so we have to shadow existing vars. This seems like a restricted version of let initially, but after a few examples the purpose of dynamic scope will hopefully be made clear. We already know we can make new vars using def, which, as I mentioned before, act like constants. The exception to that rule is when they are defined as dynamic. Functions defined with defn are stored in a var, and so they can also be declared as dynamic (the defn macro uses def internally). A var can be defined as dynamic simply by adding “:dynamic true” to its metadata. Let’s take a look at this next example:

[NOTE: Before clojure 1.3 all vars were implicitly dynamic, which means there is a lot of broken code on the internet (including the official clojure website at the time of this writing)]

The vars x, y, and sum-of-squares are all declared to be dynamic at the top of the file. The function sum-of-squares-for-x-and-y calculates the sum of squares for the vars associated with the symbols x and y. Remember x, y, and sum-of-squares refer to vars and not values, the value of the var is looked up every time this function is run. When the clojure runtime encounters a var, it first checks for any thread-local bindings, if it can’t find any it will return the root value of the var which is shared by all threads. So in the above example when we first call sum-of-squares-for-x-and-y, all of the vars referenced in the function definition defer to their root binding. Next, we push new bindings on to our thread-local stack and call the same function, and now the function uses these new values and returns a different result. The key point here is the entire call stack below the binding point is affected, not just the text within the binding block. Once we end the binding block, the values get popped off the stack and return to their root value.

In the above example, we have a function that returns an adder function for the var x just like we did earlier using a let block. Since x is a var, the value is determined at the time the function is run, not when it is created. You cannot close over a dynamic scope! In the example below I got around this by lexically shadowing the var x with its current dynamic value.

So why would you ever want to use dynamic scope over lexical? Let’s take a look at how dynamic vars are being used in the core clojure library. The stdin and stdout file handles, which many core I/O functions use (eg. println, read), are defined by the dynamic vars *in* and *out* (by naming convention, dynamic vars are wrapped in asterisks (aka earmuffs)). Maybe you want your application to print its output to some log file instead of stdout. We can simply dynamically re-bind the *out* var at the base of our application.

Now the println function (and any other function that uses *out*) will print to log.txt. Let’s try something a little more complicated. Say we have a multi-threaded application and we don’t want the print output from each thread to be intertwined with each other. One way we could accomplish this is to have each thread print to its own log file. Since each thread maintains its own stack of bindings, this is trivial to do.

Now each threads’ print statements will go in to their own files. Note that it is rare to directly make java threads like this in clojure since clojure already has more expressive concurrency primitives (agents, futures, pmap, etc). However, since those all use threads internally these same binding rules apply which can lead to unexpected results. Some other use cases of dynamic scope include the following:

  • You can debug function calls by dynamically overriding that function to print every time it gets called. Checkout clojure/tools.trace for the excellent dotrace macro.
  • The clojure/java.jdbc library uses a dynamic var *db* so that you don’t have to pass in a database connection to every function call. You can just use the with-connection macro to bind *db* to the database you want to use, and then perform any database queries within the body of the macro.
  • It allows you to mock functions that have side-effects to make them more unit-testable.
  • If you look at the clojure core library you will find a lot of other use cases. For example, the current namespace is stored in *ns* and you can use the in-ns macro to temporarily bind that to some other namespace and dynamically add functions to the newly bound namespace.

Conclusion

In general, when you think you need a dynamic var you probably don’t. Lexical scope is easier to understand because the programmer can determine the resolution of the symbols by looking at the code. Using dynamic vars also breaks referential transparency, which greatly reduces code readability and testability. However, clojure is a practical language that allows the programmer to come up with creative and expressive solutions to problems, and recognizes that sometimes using dynamic scope might be the most straightforward way to solve a problem.