Software Engineering

Posts related to the many elements that are Software Engineering. From planning, to design, to implementation, this has it all.

Node.js vs Python vs PyPy – A Simple Performance Comparison – Updated

n_queens_graph
n_queens_table

Some History

This is a followup to my original post: Node.js vs Python vs PyPy – A Simple Performance Comparison.  This article corrects a discrepancy caused by a slight difference in the JavaScript implementation which skewed the Node.js results.

The Algorithm

As stated in the previous article, I’ve attempted to implement the same single-thread, brute force, recursive algorithm in many different languages.  There is nothing overly special about this algorithm and I’ve made no attempts to optimize it.

The Findings

Node is fast, very fast.  It easily outperforms any of the other implementations I’ve included in the puzzle’s repository.  As you can see by the included charts, the performance difference between Node.js and out-of-the-box Python is very significant and the difference between it and PyPy while less pronounced is significant.

Special Notes

  • I’ve placed my source on GitHub at the following url: https://github.com/chaddotson/puzzles.  It now contains functional N-Queens puzzle implementations in JavaScript, Python, Lua, Scala, and Ruby.  There is also a version in Rust, but that needs to be updated to the latest syntax before it can be run again.
  • This is just with one type of algorithm, the best solution might and probably does change depending on what type of application you are researching.  For webserver performance, Node.js is slightly better than PyPy running Tornado.
  • This algorithm is a simple brute force algorithm, there are many faster and better ones out there.
  • See the original article for the Node.js vs Python vs PyPy – A Simple Performance Comparison for more details memory performance.
Posted by Chad Dotson in Misc, Programming, Software Engineering, Technology, 5 comments

Notes On Writing Testable JavaScript Vol 1

When writing JavaScript, I am a big fan of minimizing functionality and variables exposed publicly, which we all know to be good practice. However, this leads to anonymous functions and functions hidden within closures.  So….

How Do You Test That?

How exactly do you test private methods in JavaScript?  To answer that you should ask yourself, should I even be testing them independently or can I write tests for the exposed functionality and still achieve code coverage?  If the answer to that question is “yes,” write tests for the exposed functionality that inherently test the underlying private functions and stop there.  If the answer is “no, I really need to test this function.”  There are a few approaches.

Member Variables For Testing Only

This approach involves creating member variables intended for testing and testing alone.  This method relies on the build process to remove the variables before going to production.  While this process works, I believe it has a code smell to it.  You are polluting and bloating the code base with needless variables.  If you are interested in the approach, here is an article about it.

The Real Question: Should It Be There?

Is the fact that you are asking this question an indicator of a code design issue?  Perhaps the code is in violation of the Single Responsibility Principle?  I’ve recently experienced a little epiphany associated with this.  I realized that a collection of private functions that I was hiding actually belonged to a separate object as public functions.  This refactoring drastically reduced the code complexity, made it more maintainable, and enabled small, important functions to be separately tested.

 

 

 

 

 

Posted by Chad Dotson in Programming, 1 comment

Make It Easy

Building a successful product is usually complicated business.  With any luck a project will have an automated deployment process.  This however is only part of the equation.  Another significant part would be an automated build process.

Long Term Success

Long term success means making it easy for new people to get started in the weeks/months/years following a project’s startup.  Imagine the following project in two different scenarios.

The project is a large scale application with several dependencies.

Scenario 1 (No automated configuration and build process):

  1. Check out project from source control.
  2. Perform configuration needed for dependencies.
  3. Build / Install each dependency separately.
  4. Perform configuration needed for product build.
  5. Build product.

Scenario 2:

  1. Check out project from source control.
  2. Build product.

Which of those scenarios is more straight-forward and easiest to work with?  It’s pretty easy to see that scenario 2 is the best.

Memory and Documentation

In addition to helping new team members get started,  automated builds can serve as a form of long term memory.

“How do I do that?” becomes “press build.”

“How does that work?” becomes “check the build script.”

Posted by Chad Dotson in Key Concepts, Programming, Software Engineering, 1 comment

Machine Learning and the Curiosity Engine

The Future

As we progress technologically, there is an increased focus on machine learning.  That is a machine that is not necessarily programmed with the algorithm to solve a problem, but one that has been programmed with the ability to take a given set of input, classify it, and generate an answer.  This is achieved via supervised, unsupervised, or reinforced learning techniques.  My background in this field is creating neural nets and I am more that a bit rusty so for more on topic of machine learning see wikipedia.  I believe machine learning to be only half of the equation to achieve true AI.

Achieving True AI

Machine learning still requires a human to create the learning data set, expected results, and training algorithm.  This is an incomplete view of what is necessary to achieve true artificial intelligence.

  • We must teach machines how to learn a new general task using its current capabilities and apply that to future tasks.
  • We must teach machines how to incorporate new capabilities and algorithms into themselves.
  • We must teach machines a process to learn new tasks with little or no user input.
  • Above all, we must teach machines how to be curious.

The Curiosity Engine

As living, breathing creatures it is our curiosity that drives us to learn new things.  It is how we learn to do anything new from walk to drive to fill-in-the-blank.  I believe that Machine Curiosity is critical to the future of AI.  Once we figure it out, the possibilities are endless.  So this raises a good question, how do we teach machines to be curious?  I don’t think that anyone really knows the answer to that question.  Here, however, are some possibilities.

  • Mimicking –  While mimicking is a learning method, maybe it can help with curiosity too.  The machine could analyze behavior, identify behavior that it doesn’t know, then attempt to mimic the behavior.  This could be considered a crude form of machine curiosity by itself.
  • Chaos (random combinations of known abilities) – Another crude form of curiosity. The machine simply pairs capabilities in the attempt to do something new.  This is brute force, and might take along time to create useful abilities.
  • Observation – The machine selects observed capabilities and applies its current capabilities to it in order to learn about the expected results.  The machine would need a way to create new analytical capabilities.
  • …  Many many more methods …
  • A mix of the above – All of the above can be combined into one curiosity engine.  This method has the potential to be the most efficient since it will make intelligent use of all other methods.
Posted by Chad Dotson in Key Concepts, Technology, 0 comments

Becoming an Entrepreneur as a Software Engineer Vol 1

This is the first post in what I plan to be a series.  In these posts, I will explore ideas in entrepreneurship.  It’s a given that working for yourself, while it brings many risks, can bring many rewards.  I believe that it is one of the only ways to achieve freedom from the daily grind and to meet long term financial goals.

Benefits Of Working For Yourself

  • Achieve – The sky is the limit.
  • Vision – Your the one with the vision (for the company and/or product).
  • Destiny – You make your own destiny.
  • Freedom – You call the shots.
  • Profit – As owner, you reap the rewards.

What I Think Works

My thoughts on what works will probably change a lot over time, but currently I think one of the best ways to become a successful entrepreneur is to develop applications, websites, devices, etc that are an improvement to a larger company’s product in the hopes that they buy out your company.  For example, $10 million is a lot for an individual working the daily grind, but pocket change to large corporations.  Buying other companies is at the heart of how Apple, Google, Microsoft, Facebook, or any other major companies acquire new technology and features so it definitely works for some people.  Think of whatever your company is bought for as excellent seed money into the next.  The big “but” here is don’t undersell either.  If you can see a larger valuation for your company/product, stick with it and don’t take a buyout.

What Doesn’t Work – Writing Books

Apparently, writing books isn’t as profitable as you would think.  If you listen to the Entreprogrammers podcast, you’ll hear just how much they make on their books.  While I don’t remember the exact numbers they say, it wasn’t a lot (only a fraction of a good year’s salary).  It’s enough to make me think that unless you are some big publishing company its simply not worth your time and effort.

Posted by Chad Dotson in Programming, Software Engineering, Work, 0 comments

My Swift App – An Ongoing Experiment

Work is progressing on my first iOS app written in Swift.  I think that I might have a rudimentary alpha product ready to start testing in a few weeks.  Based on my experience with it so far, here are some of my thoughts, observations, and concerns.

  • Developing in Swift is as the name implies, “Swift.”  I didn’t know anything about Objective-C nor Swift before I started my app and I think I’m making decent progress given the amount if time I’ve had to work on it.  Given proper resources, I believe its possible to progress Swift apps from concept to production in a matter of weeks.
  • Language resources (documentation and examples) are still pretty sparse.  Get ready to learn a little Objective-C if you need help with API calls because that is where the majority of the help you will find will be.
  • The iOS 8 Simulator doesn’t save location privacy settings,  That means that every time you run an app that requires location privileges, you have to edit the settings again.  Based on what I’ve read, this issue has existed for some time in the beta code.
    This was due to user error.  You have to use either the location manager’s requestWhenInUseAuthorization or requestAlwaysAuthorization functions paired with the right entries in your projects Info.plist file.  The reason mine wasn’t working was I had erroneously edited the plist file associated with the unit tests.

    <key>NSLocationWhenInUseUsageDescription</key>
    <string>Use location when open?</string>
    <key>NSLocationAlwaysUsageDescription</key>
    <string>Use location always?</string>
  • Perhaps the biggest issue is that Xcode cannot refactor Swift code.  To me this seems like a giant shortcoming of the editor.  Hopefully Apple will push an update soon to correct this.
  • While storyboards are powerful, some of the functionality is not so discoverable.  The auto-layout functionality took me awhile to find and not without searching the net.  I think they maybe onto something, but I do tend to like the way Visual Studio makes similar functionality “findable” from the properties panel.
  • As of Xcode 6.0.1 and iOS Simulator 8.0, AVSpeechSynthesizer doesn’t appear to work at all.
    var mySpeechSynthesizer:AVSpeechSynthesizer = AVSpeechSynthesizer()
    var myString:String = "This is a test."
    var mySpeechUtterance:AVSpeechUtterance = AVSpeechUtterance(string:myString)

    When executed in the iOS simulator, the code results in “Speech initialization error: 2147483665”.  I’ve seen a few work arounds on the net, but I don’t believe any of them actually work in Swift.

Posted by Chad Dotson in Programming, Software Engineering, 2 comments

Swift – A Quick First Impression

Swift and Xcode 6

Late last week Apple released Xcode 6 to the general public.  I’ve been waiting for the opportunity to try out Swift, so I started working on an idea I’ve had.

General Thoughts

This is probably a general iOS development comment, but the way you link items on the storyboard to class members seems odd to me (having done a lot of C# development in the past).  However, it does have its advantages.  Counter to the Microsoft environment, it does seem to encourage better design and discourage the worse of some bad habits.

Compared to Android

Its been awhile since I worked with Android, but when I did developing for the platform was disastrous.  Setup of the environment (Eclipse) and the emulator was time-consuming and not so straight-forward.  With Xcode, its all on rails.  You can have a “Hello World” app up in the simulator in 2 minutes.

Problems

The language does seem to be a little wordy at first glance maybe that opinion will get better as I get more experience with it.  While the official Apple documentation is now complete with Swift equivalents to Objective-C,  examples both official and third party are pretty few and far between it seems.

To be continued…

Posted by Chad Dotson in Programming, 0 comments

DSLR Lightning Trigger

June 15, 2011 Lightning CompositeLightning

I use my Nikon D5000 to take a lot of lightning photos.  This one to the right is a composite of several taken during a storm that produced a good amount of of photogenic lightning a few years back.  The photos that make up this one and many of my others were created using timed exposures of up to 30 seconds on length.  This is problematic, at times, due to other light sources, camera movement, etc.  Using simple, arduino-based circuit, we can create a device that will attempt to detect lightning strikes and trigger a camera shutter when one is detected.

Prototyping The Circuit

This circuit consists of 2 major pieces: lightning detection and shutter triggering.  To detect lightning, a phototransistor or photoresistor is required.  For this project, I used a photoresistor.  To trigger the shutter, a 940nm infrared LED is required.  These LEDs can be used to trigger cameras from Nikon, Canon, Pentax, Olympus, Minolta, Sony and possibly others.  Below is the complete parts list I used along with links to them on Amazon.

Components:

  • Arduino UNO R3 (buy)
  • Photoresistor 5mm GM5539 (buy)
  • 940nm Infrared LED (buy)
  • Yellow LED (buy)
  • 1 – 200 Ω Resistor (buy)
  • 1 – 100k Ω Resistor (buy)
  • Miscellaneous jumpers and breadboard (buy)

 

The Code

This circuit’s method of detecting lightning with a photoresistor is pretty simplistic.  It simply loops infinitely recording the analog input from the photoresistor, taking the difference of it to the saved value, and comparing that to the configurable threshold.  In its current configuration, it will take 2 photos per lightning strike.  This is the first iteration of this code, so I may change it to use a moving average instead of just the last recorded value.

To fire the IR LED in the right sequence, I used the Multi-Camera IR Control Library since it supports my Nikon D5000.  In addition to the D5000, this library supports Nikon, Canon, Pentax, Olympus, Minolta, and Sony cameras. If you don’t have a camera that is supported, a custom interface could easily be created with specifics for your camera model.

#include <multiCameraIrControl.h> 

int shutterPin = 13; 
int triggerPin = 0;
int threshold = 5;
int savedLightningValue = 0;
int currentLightningValue = 0;
int delayBetweenShots = 1000;
 
Nikon D5000(shutterPin);

void setup() {
  Serial.begin(9600);
  pinMode(shutterPin, OUTPUT);
  digitalWrite(shutterPin, LOW);
  savedLightningValue = currentLightningValue = analogRead(triggerPin);
}

void loop() {
  currentLightningValue = analogRead(triggerPin);
  Serial.println(savedLightningValue, DEC);
  
  if(abs(currentLightningValue - savedLightningValue) > threshold) {
    Serial.println("Triggering shutter");
    D5000.shutterNow();
    delay(delayBetweenShots);
  }
  savedLightningValue = currentLightningValue;
}

 

To The Field

Since the protoype is fully functional, I plan on fielding it during the next thunderstom.  I will report back once I see how it works along with any refinements.

Resources

As always, I have placed the code and diagrams for this project in a github repository.  It includes code that can be compiled and uploaded via the Arduino IDE and diagrams that can be viewed or edited with Fritzing.

Other Posts in this series:

Posted by Chad Dotson in Arduino, Hobbies, Photography, Programming, Raspberry Pi, Software Engineering, Technology, 8 comments