Software Engineering

Posts related to the many elements that are Software Engineering. From planning, to design, to implementation, this has it all.

Aug

2014

Comparing Node.js and Python on the Raspberry PI

Picture of my simple led circuit connected to the Pi.

Programming on the Raspberry Pi

Python seems to be the more popular language for writing programs on the Raspberry Pi. However, it is far from the only language: Python, C, C++, Java, and Ruby are some that are automatically supported out of the box. As it turns out Node.js is also supported. Perhaps, just based on my preconceptions, I didn’t originally consider JavaScript when interfacing directly with hardware. The goal of this article is to compare functionally equivalent sample programs written both in Python and JavaScript.

The Goal

I decided to make the scope of the problem created for this article as small as possible. Given that the goal will be to create simple circuit diagram consisting of a LED and then to write a program that makes the LED flash for a specified period of time.

Simple LED circuit diagram

Hardware

With the Pi, a circuit that will make a LED flash is relatively simple: Consisting of just a LED and resistor. I arbitrarily selected pin 7 on the Pi for this build. I’ve included a diagram of it on this page.

Software

As the goal states, the following two implementations do nothing except toggle an LED at half the specified duration (0.5s) and automatically shutoff after 60 seconds.

Python

Since Python is the more traditional programming language for the Raspberry Pi, let us start with it. The Rpi.GPIO module I used can be installed via pip. Before you read the code sample, let me point out some things about the implementation. Could it have been done simpler given that the problem was just to make the LED flash for a certain amount of time? Yes, but I wanted to create something of an asynchronous process that could be controlled from outside its execution context.

import RPi.GPIO as GPIO

from threading import Event, Thread, Timer

class ImprovedThread(Thread):
    def __init__(self, *args, **kwargs):
        super(ImprovedThread, self).__init__(*args, **kwargs)
        self._stopEvent = Event()

    def stop(self):
        self._stopEvent.set()

    def is_stopped(self):
        return self._stopEvent.isSet()

    def wait(self, duration):
        self._stopEvent.wait(duration);


class LEDFlasher(ImprovedThread):
    def __init__(self, pin, duration):
        super(LEDFlasher, self).__init__(target=self._flashLED)
        self.setDaemon(True)

        self._pin = pin
        self._duration = duration
        self._state = False

        GPIO.setmode(GPIO.BOARD)
        GPIO.setup(self._pin, GPIO.OUT)

        self.start()

    def _flashLED(self):
        while not self.is_stopped():
            GPIO.output(self._pin, self._state)
            self._state = not self._state
            self.wait(self._duration/2)

flasher = LEDFlasher(7, 0.5)
Timer(60, flasher.stop).start()

flasher.join()

GPIO.cleanup()

JavaScript

I used pi-gpio to interface with the GPIO on the Pi in JavaScript. It can be installed via npm. It makes use of gpio-admin so that your script doesn’t have to run as sudo. Follow the installation instructions provided here and it should get you setup. Note that due to differences int the languages, it was not necessary to thread (web workers) the solution.

var gpio = require("pi-gpio");

function flashLED(pin, duration) {
    return setInterval(function() {
        gpio.open(pin, "output", function(err) {
            gpio.write(pin, 1, function() {
                setTimeout(function() {
                    gpio.write(pin, 0, function(err) {
                        gpio.close(pin);
                    });
                }, duration/2);
            });
        });
    }, duration);
}

var intervalID = flashLED(7, 500);

setTimeout(function() {
    clearInterval(intervalID);
}, 60000);

The Findings – I was surprised

When I started writing the JavaScript side of this article, I was mainly doing it to gain experience working with JavaScript on the Pi. I did not expect to come out of it vastly preferring it over my python implementation. I find the Python implementation above to be to wordy and overcomplicated. It just seems that the amount of code needed to achieve the same results is excessive. Perhaps this is because of the problem scope and implementation. The problem posed here was a simple one, where a functional solution is superior to an object-oriented one. Could the Python code above be rewritten into something a bit more functional? Sure! Are there problems that an object-oriented Python or JavaScript implementation would be the better solution for? Definitely. The lesson to take away: be open to working outside your comfort zone and pick solutions to problems based on their fit for the problem not your comfort level with them.

I’ve created a copy of the code as a Gist that is available here: https://gist.github.com/chaddotson/570501a3e3dcfe8928c8

Aug

2014

Everyone’s A Coder

Question:

Are we headed to an era where everyone should know how to write at least some rudimentary code?

Learn to Code

Today, there are many sites on the internet offering to “teach you to code.” Two of which are KhanAcademy and CodeAcademy. These sites offer a selection of topics that can be worked through in the span of a few hours. Can it be done? Sure. Matter of fact, I think they are good resources to get you started.

Differentiating Computer Science

Computer science is more than just coding. Wikipedia defines computer science as the following:

Computer science is the scientific and practical approach to computation and its applications. It is the systematic study of the feasibility, structure, expression, and mechanization of the methodical procedures (or algorithms) that underlie the acquisition, representation, processing, storage, communication of, and access to information, whether such information is encoded as bits in a computer memory or transcribed in genes and protein structures in a biological cell. A computer scientist specializes in the theory of computation and the design of computational systems.

Simply put, Computer Science is more than programming, though most computer scientists do write code. On the topic of code, colleges don’t so much focus on languages as they do the theory, math, and algorithms. This allows graduates to quickly assimilate new languages and ideas.

Does a product (or code) have to be perfect?

The answer is a pretty resounding “No,” but bad or inefficient design might harm long term viability. In terms of a startup, I’d say a Software Engineer should become involved soon after completion of the MVP if not before.

Can everyone be a coder?

Yes! I believe that in the future everyone should know how to write at least some code. Right now, JavaScript and Python are the best candidates that could be learned by all. With it leaning in JavaScript’s favor with Node.js. Though, if you have serious aspirations to getting a career writing code, I suggest a formal degree.

Aug

2014

Project Excelsior – Technology Update

Current Project Excelsior Technology

For the past month or so I’ve been working on a little side project that I’ve dubbed “Project Excelsior.” I’ve not had an abundance of time to spend on it over the past few week, so I’m just now wringing out some of the technology stack. I’ve almost decided completely on the MEAN stack. I’m close enough to a final decision that I’ve started prototyping the server side. I’ve not worked with angular before, but I am familiar with ember and backbone.

Aug

2014

No Comments – A Failure Twice

Everyone that writes code has encountered or written their fair share of undocumented code.

A failure twice?

I was encouraged to write this article because of something I read about the pinball game that once was included with windows. According to the article on MSDN, the pinball game was removed due to a bug that should have been fixable. However, the overall qualify of the code made repair impossible and the game was removed from the distribution. Two major items can be taken from this:

Failure 1: The could should have been self-documenting / few comments required.
Failure 2: Code not easily understood should have been commented.

Properly commented code can be tricky!

In college they push you to comment your code while not stressing that over documentation is also bad. I remember writing programs that had comments on almost every line for assignments. In the end it doesn’t buy you anything, it just restates the obvious and clutters the solution. In the workplace, comments and whether or not the code needs them are hit and miss.

Some notes about comments:

Many comments are an acknowledgement of your failure to communicate. Write better self-documenting code.
Outdated or wrong comments are worse than no comments.
Consider refactoring code that’s not self-documenting. I find one of the biggest places this can be done is extracting methods from if statements.
Choose a good, descriptive names. I’m a little wordy in my names, but in the end most of my code can almost read like a sentence.

Remember

The code is your best documentation. Think about what you want to communicate with it and how to be as clear as possible.

Aug

2014

Node.js vs Python vs PyPy – A Simple Performance Comparison

IMPORTANT NOTE: The NodeJS algorithm had a slight discrepancy in it. See this article for a correction to the performance comparison section of this article.

The Algorithm

Yesterday, I decided to try translate my algorithm for calculating N-Queens to JavaScript. I’ve implemented the same single-thread, brute force, recursive algorithm in many different languages with the biggest difference being the syntax of the language. Once I completed the JavaScript Implementation, I ran the program with the latest version of Node.js.

The Findings

I knew Node was fast but it still surprised me. As you can see by the included charts, the performance difference between Node.js and out-of-the-box Python is pretty significant. Its not until the algorithms complexity and recursion depth hit certain limits that Node.js’s performance starts to falter.

Node.js and CPython – What’s The Difference?

You might ask what is behind this performance difference. The answer is actually pretty simple. It all boils down to how the code is being executed. Node.js uses the V8 JavaScript Engine (Wikipedia | Google) written by Google and a part of the Chrome Browser. V8 includes a just-in-time compiler that compiles the JavaScript to machine code before execution and then continuously optimizes the compiled code. Python is a bytecode interpreter; meaning that the default interpreter (CPython) doesn’t execute Python scripts directly. Instead, it first generates a intermediate file that will later be interpreted at runtime.

Ways To Get Better Performance

If you want to use Python, we can overcome the differences between Node.js and vanilla Python by using PyPy, an alternative implementation of Python that includes a just-in-time compiler. For the algorithm I wrote, you can see a pretty good performance boost over Node.js when using PyPy.

Special Notes

I’ve placed my source on GitHub at the following url: https://github.com/chaddotson/puzzles
This is just with one type of algorithm, the best solution might and probably does change depending on what type of application you are researching. For webserver performance, Node.js is slightly better than PyPy running Tornado.
This algorithm is a simple brute force algorithm, there are many faster and better ones out there.
At a board size of 15, Node.js could no longer run the algorithm due to its maximum recursion limit.

Edit – A Follow-Up

The original focus of this article was shear performance, but I’ve received a question regarding the memory footprint of the 3 methods. I think that is a very good and valid question. So, I reran the tests to capture the peak memory utilization by each. For this test I used “/usr/bin/time -l” to capture the maximum resident set size. While this isn’t exactly the peak amount of memory utilized by the algorithm, it is sufficiently close to report on.

New Findings

Upon rerunning the tests for capturing memory utilization, I found that for the most part memory utilization contrasts performance. A higher memory utilization isn’t really unexpected, if you think about it. Essentially, the jit is sacrificing memory for performance. In most cases, this isn’t really that bad. Using a jit is just a cheap way of boosting performance of code written in an interpreted language. The boost in performance, speed of which it was written and the maintainability of it outweigh memory utilization concerns in many cases.

The Oddity

As you can see, I’ve included a chart covering all the solutions for boards 8×8 to 14×14. During most increments in board size, the memory utilization seems to increase exponentially; however, when we hit the 14×14 board size we see all the cases level off at relatively the same memory utilization of around 300 MB. At this time, I really don’t have a good answer for this. I could certainly speculate, but I’d rather not until I know more.

Jul

2014

Knowing When You’ve Wrote Crappy Code

Note: This article was kicked off by one I read over at LosTechies.

We write lots of code, statistically speaking some of it is what we’d deem as “crappy.” If you’re doing things right and progressing in your career and your understanding; your definition of “crappy code” should change over time. This is very important concept for a good Software Engineer, its one of the ways we get better. We recognize our past coding mistakes and work to better them.

Dangers of Crappy Code:

It could hide technical debt. “I don’t really know how or why this works, but here it is. Done.”
You’ve not taken the time to make what works, right. This potentially leads to bloated code, duplication, and an overall poor product quality.
Is it robust?
It is, potentially, not very reusable.
Can I hand this code off to someone else and they understand it?

Preventing Crappy Code:

The prospect of writing crappy code should not prevent you from getting a project working. However, it is very important to make it as right possible before committing the code changes to the repository.

Write self-documenting code. Some developers may scoff at this statement, but self documenting code is very possible in just about any language. Remember comments are your failure to communicate.
Remember to refactor, refactor, and refactor.
Ask for the opinion of a peer. This should be someone you consider qualified enough to give an opinion.
Use “TODO” comments so your thought process is not lost and it will serve as a reminder that you must still make something right. This doesn’t prevent crappy code from making it into a project but it documents its existence.

What To Do When You Find Crappy Code:

For the sake of this topic, lets assume you run across some crappy code 2 years into maintenance of a software product.

Identify why it is “crappy.”
Does it work or has it unknowingly introduced bugs into the system?
Perhaps leave a NOTE comment in the code, especially if it can add insight into the function of the crappy code.
Only change the code if it is within the scope of your current task. Remember that it has worked for 2 years and changing it now could potentially introduce error. If you do end up changing the code, attempt to make it right before you finish. Also make sure to update your unit tests.

Are you Ready? A Simple Test

Review a code base that you wrote 2 years ago. Did you find “crappy code?” The answer should almost certainly be a “Yes.”

Jul

2014

Isolate, Understand, Implement

Isolate, understand, implement: three very important things to remember when adding, modifying or replacing features on a project. The first step is to isolate the feature. Hopefully, your project is designed in such a way that once the code is isolated it is in a single place. The next step is understand. Meaning understand what the code currently does. Run the unit tests. Make sure you know without a doubt why the code works and what the code does. Also understand any effects that your changes might have. And finally, implement the new code or changes. During this phase you’ll modify the unit test set as appropriate, make the necessary changes to the code base, and ensure that the application is stable before pushing it back to your repository.

Jun

2014

Generating JSON Documents From SQLite Databases In Python

Special Note

This article assumes that you do not wish to use a more sophisticated ORM tool such as SQLAlchemy.

Some Setup

Let’s start with a Q&D sqlite database given the following sql.

create table sample(column1 INTEGER, column2 TEXT, column3 TEXT, column4 REAL);
insert into sample(column2, column2, column3, column4) values(1, "Record 1 Text A", "Record 1 Text B", 3.14159);
insert into sample(column1, column2, column3, column4) values(2, "Record 2 Text A", "Record 2 Text B", 6.28318);
insert into sample(column1, column2, column3, column4) values(3, "Record 3 Text A", "Record 3 Text B", 9.42477);

You can create the sqlite database given the following command.

$ sqlite3 sample.db < sample.sql

Some Different Methods

For this example we want each record returned via the sql select statement to be its on JSON document. There are several ways of doing this. All of them solve the problem reasonably well but I was in search of the best way. In checking python.org, I discovered that the sqlite connection object has an attribute falled row_factory. This attribute can be modified provide selection results in a more advanced way.

Method 1 – My Preferred Method

From the python docs, we find that they already have a good factory for generating dictionaries. It is my opinion that this functionality to should be more explicitly enabled in the language.

In this method, we override the row_factory attribute with a callable function that generates the python dictionary from the results.

# https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.row_factory

import sqlite3

def dict_factory(cursor, row):
    d = {}
    for idx, col in enumerate(cursor.description):
        d[col[0]] = row[idx]
    return d

con = sqlite3.connect(":memory:")
con.row_factory = dict_factory
cur = con.cursor()
cur.execute("select 1 as a")
print cur.fetchone()["a"]

Method 2 – Almost As Good As Method 1

This method is just about as good as method 1. Matter of fact, you can get away with this one and be just fine. Functionally, the methods are almost identical. With this method, the records can be accessed via index or via column name. The biggest difference is that unlike method 1, these results don’t have the full functionality of a python dictionary. For most people, this might be enough.

con = sqlite3.connect(":memory:")
con.row_factory = sqlite3.Row
cur = con.cursor()
cur.execute("select 1 as a")
print cur.fetchone()["a"]

Putting It All Together

The following code snippet will extract a group of dictionaries based on the select statement from the sqlite database and dump it to JSON for display.

The Code

#!/bin/python

import sqlite3

def dict_factory(cursor, row):
    d = {}
    for idx, col in enumerate(cursor.description):
        d[col[0]] = row[idx]
    return d

connection = sqlite3.connect("sample.db")
connection.row_factory = dict_factory

cursor = connection.cursor()

cursor.execute("select * from sample")

# fetch all or one we'll go for all.

results = cursor.fetchall()

print results

connection.close()

The Results

[
    {
        "column1": 1,
        "column2": "Record 1 Text A",
        "column3": "Record 1 Text B",
        "column4": 3.14159
    },
    {
        "column1": 2,
        "column2": "Record 2 Text A",
        "column3": "Record 2 Text B",
        "column4": 6.28318
    },
    {
        "column1": 3,
        "column2": "Record 3 Text A",
        "column3": "Record 3 Text B",
        "column4": 9.42477
    }
]

Programming on the Raspberry Pi

The Goal

Hardware

Software

Python

JavaScript

The Findings – I was surprised

Question:

Are we headed to an era where everyone should know how to write at least some rudimentary code?

Learn to Code

Differentiating Computer Science

Does a product (or code) have to be perfect?

Can everyone be a coder?

A failure twice?

Properly commented code can be tricky!

Remember

IMPORTANT NOTE: The NodeJS algorithm had a slight discrepancy in it. See this article for a correction to the performance comparison section of this article.

The Algorithm

The Findings

Node.js and CPython – What’s The Difference?

Ways To Get Better Performance

Special Notes

Edit – A Follow-Up

New Findings

The Oddity

Dangers of Crappy Code:

Preventing Crappy Code:

What To Do When You Find Crappy Code:

Are you Ready? A Simple Test

Special Note

Some Setup

Some Different Methods

Method 1 – My Preferred Method

Method 2 – Almost As Good As Method 1

Putting It All Together

The Code

The Results

Recent Posts