You’re Too Close

Have you encountered the following scenario?

You are trying to solve a problem (or helping solve a problem) and know or at least think you know the solution.  You are in the middle of implementing it when someone else looks at it and says, “why don’t you do it this way, isn’t this way easier/better?”  Taking a step back, you realize that the question not only has merit but is a better and much more obvious solution; you can’t believe you missed it.

What happened?

I think its because you were too close to the problem and had developed a very narrow focus.  That narrow focus prevented you from seeing the better solution.  Perhaps this is even a variation of functional fixedness in that we’ve latched onto an idea of how to solve a problem and our mind’s may not see alternatives easily.

What can we do?

  • Think about the broad (or product) level goals regularly.
  • Entertain questions and/or suggestions from others.
  • Ask: “Is this the best way?”
  • Ask: “Is this the practical way?”
  • Don’t overthink the problem.
  • Get it working then evaluate the solution and/or do a code review!
Posted by Chad Dotson in Doing Things Better, Programming, Tips, 0 comments

Python Logger

The Scenario

This scenario illustrates two possible mistakes people make when using the python logging module.  Analyze the following code and look for issues.

So what is wrong with that?

First and foremost, the code fails to use the existing Logging.exception function that could and in most cases should be used when logging exceptions.  That function will automatically add all the exception info to the log, meaning that you will have the stack trace!  Secondly, this sample used the string.format function to format the log message for the logging library when the logging library can in fact handle string formatting itself via old style format specifiers.

Fixing it

If I were to ignore the first problem, the following code is what I should have written.  The benefit here is that the formatting is only executed if the log message is to be captured, unlike the first method.

Taking both errors into account, we should have used the exception function instead of the error function on the logger as well as the built in formatting.  Given both of these, the code becomes.

More Data

This scenario led me to quantifying the error in execution time.  The first set of data is related to logging alone; the second set extends to timing the different string formatting options.  As you can see by the data, using the format is a good bit slower than the built-in “old-style” formatting in the logging package.  While it will add up, it isn’t a world ending difference if done on a small scale.  Again, the time difference is largely due to the fact that no formatting takes place unless the message has a high enough level.  This data caused me to extend my study into timing the two different formatting options.  As you can see by the data, the “old style” is marginally slower than the format style.

Comparing old style to new style string formatting


In the end

You should use functionality the API gives you.  In most cases, and the case with python, it has been engineered to work, be fast and be maintainable.  For more information on the logging module, check the python docs.  2.7 or 3.5.

Posted by Chad Dotson in Doing Things Better, Programming, Software Engineering, Technology, Tips, 0 comments

C-Style Unions And Python

So, you’re creating a C Union (used to create a variant type) and writing it to a file, socket, etc and want to read that in Python. There are two ways to deal with this.

Assume the following union definition in C

In C, reading the value represented by this is easy.  Since its 4 bytes, you simply read 4 bytes and then reference the appropriate element.  In Python, if your looking for functionality to closely match C, it seems not so straight forward.

struct.pack and struct.unpack

The first thing you try to do is look at the struct module and see if pack and unpack can come close to doing what you want.  The problem with pack and unpack is that it requires a data type.

This works just as well as anything and is completely straightforward, the big problem here is speed.  First, we have to do an if around each call to unpack to get the appropriate option.  Second, its faster to pull in arrays in python than single values.

A ctypes addition to struct.pack and struct.unpack

Using ctypes, you can approach a functionality similar C.  Take the following code for example.

Notice that it is always unpacking the data as an integer into the integer part of the union.  This approach has a few advantages.  One, it functions the same as the C version of the code would.  Two, you can unpack entire arrays at once which can be faster.


The first code sample seems to be the simplest and most straightforward though potentially slower.  However, depending on the situation, you may want an implementation similar to the second.

Posted by Chad Dotson in Programming, 0 comments

Node.js vs Python vs PyPy – A Simple Performance Comparison – Updated


Some History

This is a followup to my original post: Node.js vs Python vs PyPy – A Simple Performance Comparison.  This article corrects a discrepancy caused by a slight difference in the JavaScript implementation which skewed the Node.js results.

The Algorithm

As stated in the previous article, I’ve attempted to implement the same single-thread, brute force, recursive algorithm in many different languages.  There is nothing overly special about this algorithm and I’ve made no attempts to optimize it.

The Findings

Node is fast, very fast.  It easily outperforms any of the other implementations I’ve included in the puzzle’s repository.  As you can see by the included charts, the performance difference between Node.js and out-of-the-box Python is very significant and the difference between it and PyPy while less pronounced is significant.

Special Notes

  • I’ve placed my source on GitHub at the following url:  It now contains functional N-Queens puzzle implementations in JavaScript, Python, Lua, Scala, and Ruby.  There is also a version in Rust, but that needs to be updated to the latest syntax before it can be run again.
  • This is just with one type of algorithm, the best solution might and probably does change depending on what type of application you are researching.  For webserver performance, Node.js is slightly better than PyPy running Tornado.
  • This algorithm is a simple brute force algorithm, there are many faster and better ones out there.
  • See the original article for the Node.js vs Python vs PyPy – A Simple Performance Comparison for more details memory performance.
Posted by Chad Dotson in Misc, Programming, Software Engineering, Technology, 5 comments

Notes On Writing Testable JavaScript Vol 1

When writing JavaScript, I am a big fan of minimizing functionality and variables exposed publicly, which we all know to be good practice. However, this leads to anonymous functions and functions hidden within closures.  So….

How Do You Test That?

How exactly do you test private methods in JavaScript?  To answer that you should ask yourself, should I even be testing them independently or can I write tests for the exposed functionality and still achieve code coverage?  If the answer to that question is “yes,” write tests for the exposed functionality that inherently test the underlying private functions and stop there.  If the answer is “no, I really need to test this function.”  There are a few approaches.

Member Variables For Testing Only

This approach involves creating member variables intended for testing and testing alone.  This method relies on the build process to remove the variables before going to production.  While this process works, I believe it has a code smell to it.  You are polluting and bloating the code base with needless variables.  If you are interested in the approach, here is an article about it.

The Real Question: Should It Be There?

Is the fact that you are asking this question an indicator of a code design issue?  Perhaps the code is in violation of the Single Responsibility Principle?  I’ve recently experienced a little epiphany associated with this.  I realized that a collection of private functions that I was hiding actually belonged to a separate object as public functions.  This refactoring drastically reduced the code complexity, made it more maintainable, and enabled small, important functions to be separately tested.






Posted by Chad Dotson in Programming, 1 comment

Make It Easy

Building a successful product is usually complicated business.  With any luck a project will have an automated deployment process.  This however is only part of the equation.  Another significant part would be an automated build process.

Long Term Success

Long term success means making it easy for new people to get started in the weeks/months/years following a project’s startup.  Imagine the following project in two different scenarios.

The project is a large scale application with several dependencies.

Scenario 1 (No automated configuration and build process):

  1. Check out project from source control.
  2. Perform configuration needed for dependencies.
  3. Build / Install each dependency separately.
  4. Perform configuration needed for product build.
  5. Build product.

Scenario 2:

  1. Check out project from source control.
  2. Build product.

Which of those scenarios is more straight-forward and easiest to work with?  It’s pretty easy to see that scenario 2 is the best.

Memory and Documentation

In addition to helping new team members get started,  automated builds can serve as a form of long term memory.

“How do I do that?” becomes “press build.”

“How does that work?” becomes “check the build script.”

Posted by Chad Dotson in Key Concepts, Programming, Software Engineering, 1 comment

My Swift App – An Ongoing Experiment

Work is progressing on my first iOS app written in Swift.  I think that I might have a rudimentary alpha product ready to start testing in a few weeks.  Based on my experience with it so far, here are some of my thoughts, observations, and concerns.

  • Developing in Swift is as the name implies, “Swift.”  I didn’t know anything about Objective-C nor Swift before I started my app and I think I’m making decent progress given the amount if time I’ve had to work on it.  Given proper resources, I believe its possible to progress Swift apps from concept to production in a matter of weeks.
  • Language resources (documentation and examples) are still pretty sparse.  Get ready to learn a little Objective-C if you need help with API calls because that is where the majority of the help you will find will be.
  • The iOS 8 Simulator doesn’t save location privacy settings,  That means that every time you run an app that requires location privileges, you have to edit the settings again.  Based on what I’ve read, this issue has existed for some time in the beta code.
    This was due to user error.  You have to use either the location manager’s requestWhenInUseAuthorization or requestAlwaysAuthorization functions paired with the right entries in your projects Info.plist file.  The reason mine wasn’t working was I had erroneously edited the plist file associated with the unit tests.
  • Perhaps the biggest issue is that Xcode cannot refactor Swift code.  To me this seems like a giant shortcoming of the editor.  Hopefully Apple will push an update soon to correct this.
  • While storyboards are powerful, some of the functionality is not so discoverable.  The auto-layout functionality took me awhile to find and not without searching the net.  I think they maybe onto something, but I do tend to like the way Visual Studio makes similar functionality “findable” from the properties panel.
  • As of Xcode 6.0.1 and iOS Simulator 8.0, AVSpeechSynthesizer doesn’t appear to work at all.

    When executed in the iOS simulator, the code results in “Speech initialization error: 2147483665”.  I’ve seen a few work arounds on the net, but I don’t believe any of them actually work in Swift.
Posted by Chad Dotson in Programming, Software Engineering, 2 comments

Comparing Node.js and Python on the Raspberry PI

Picture of my simple led circuit connected to the Pi.

Picture of my simple led circuit connected to the Pi.

Programming on the Raspberry Pi

Python seems to be the more popular language for writing programs on the Raspberry Pi.  However, it is far from the only language: Python, C, C++, Java, and Ruby are some that are automatically supported out of the box.  As it turns out Node.js is also supported.  Perhaps, just based on my preconceptions, I didn’t originally consider JavaScript when interfacing directly with hardware.  The goal of this article is to compare functionally equivalent sample programs written both in Python and JavaScript.


The Goal

I decided to make the scope of the problem created for this article as small as possible.  Given that the goal will be to create simple circuit diagram consisting of a LED and then to write a program that makes the LED flash for a specified period of time.


Simple LED circuit diagram

Simple LED circuit diagram


With the Pi, a circuit that will make a LED flash is relatively simple: Consisting of just a LED and resistor.  I arbitrarily selected pin 7 on the Pi for this build.  I’ve included a diagram of it on this page.



As the goal states, the following two implementations do nothing except toggle an LED at half the specified duration (0.5s) and automatically shutoff after 60 seconds.



Since Python is the more traditional programming language for the Raspberry Pi, let us start with it.  The Rpi.GPIO module I used can be installed via pip.  Before you read the code sample, let me point out some things about the implementation.  Could it have been done simpler given that the problem was just to make the LED flash for a certain amount of time?  Yes, but I wanted to create something of an asynchronous process that could be controlled from outside its execution context.



I used pi-gpio to interface with the GPIO on the Pi in JavaScript.  It can be installed via npm. It makes use of gpio-admin so that your script doesn’t have to run as sudo.  Follow the installation instructions provided here and it should get you setup.  Note that due to differences int the languages, it was not necessary to thread (web workers) the solution.


The Findings – I was surprised

When I started writing the JavaScript side of this article, I was mainly doing it to gain experience working with JavaScript on the Pi.  I did not expect to come out of it vastly preferring it over my python implementation.  I find the Python implementation above to be to wordy and overcomplicated.  It just seems that the amount of code needed to achieve the same results is excessive.  Perhaps this is because of the problem scope and implementation.  The problem posed here was a simple one, where a functional solution is superior to an object-oriented one.  Could the Python code above be rewritten into something a bit more functional? Sure!  Are there problems that an object-oriented Python or JavaScript implementation would be the better solution for?  Definitely.  The lesson to take away: be open to working outside your comfort zone and pick solutions to problems based on their fit for the problem not your comfort level with them.

I’ve created a copy of the code as a Gist that is available here:

Posted by Chad Dotson in Key Concepts, Programming, Raspberry Pi, Software Engineering, 8 comments