Software Engineering

Posts related to the many elements that are Software Engineering. From planning, to design, to implementation, this has it all.

Python Logging – Best Practices

The python logging module offers a wide variety of logging options and handlers.  One thing missing from the documentation is when to use each level.

A quick foreword

You really should familiarize yourself with the logging package.  How to create new loggers (I find creating them by module very useful).  There are many ways to configure logging, I tend to like dictConfig from logging.config (but start off with basicConfig form logging).

A Word on Optimal Setups

I prefer to setup my logging with each module having its own logger.  This allows me to configure logging levels at a package and/or module level.  I typically do the following in each module to create a logger.

Assuming my package structure consists of the following:

– foo (package)
—– core (module)
—– bar (module)

We can configure varying levels of logging for each element, as seen in the following snippet from a dictConfig.

In this example, the root ( ” ) logger (those not configured by any other settings) reports INFO level and up messages.  With the exception of the bar module, the foo package only reports WARNING level and up messages.  The bar module is set to a more verbose DEBUG level, to show information needed for debugging.

Selecting A Log Message Level

Out of the box, there are six default logging levels recognized by the logging module, most are self-explanatory.  I’ll just make some notes about usage.  (From here on out, I’ll refer to my logging instance as logger.)

For general status messages, you should use logger.info (INFO).  For errors, use either logger.critical (CRITICAL) or logger.error (ERROR).  For all exceptions, use logger.exception (ERROR).  logger.exception will automatically include stack trace information about the exception for you in the log. When you want verbose debugging information, use logging.debug (DEBUG)

In Closing

  • Use the logging module instead of print statements.
  • Always use logger.exception for logging exceptions.
  • Favor logger.debug for verbose log statements.
  • Favor logger.info for most other log statements (with the exception of errors).
  • Don’t forget that each of the logging functions uses C-style formatting.
Posted by Chad Dotson in Doing Things Better, Key Concepts, Programming, Software Engineering, Technology, 0 comments

Creating Web Workers From Functions (No File Needed)

So, you want to create a web worker from a function, the problem is all the examples examples show creating a web worker using a separate JavaScript file or from embedded JavaScript source.  There is a way to do it and that is using Blob URLs.  In the example below, I have taken the simple dedicated worker example from mdn and modified it appropriately.

First, we assume that we have a function that we want to serve as the source of the web worker.

Next, we use that function’s source to create the web worker.

See this example in action on jsfiddle here: http://jsfiddle.net/chaddotson/jr5p3L7r/

Notes:

  • The createObjectURL function is experimental according to MDN, so that is a bit of a warning bell.  However, according to caniuse, it is mostly safe across all browsers.
  • This can be used to create shared web workers, but they will not be shareable between tabs or windows.
  • This makes it more difficult, but possible to assemble dependencies for the web worker.
  • This isn’t the officially supported way to create a web worker.
  • I think this approach to creating a worker should be used sparingly, if ever.

 

Posted by Chad Dotson in Programming, Software Engineering, Tips, 0 comments

Creating Downloadable CSV (Or Any Other) Files In Browser With JavaScript

Many of us have wanted to create downloadable content straight from the browser.  I remember several years back where it was more difficult and the solutions were not clean.  Today, it is much much more simple and clean.  Enter Blob URLs.

In this example, I have an object of arrays.  These arrays may or may not all be the same length.  The csvify function accounts for this by letting the user optionally specify a default value and whether or not the pad at the beginning or at the end for arrays that aren’t the same size as the max.

It’s what comes after the csvify function that allows all this to work.  To make this csvified data downloadable, I first create a blob.  This blob contains the csv data and is created with the csv mime type.  I then create a url from that blob using createObjectURL and set that url as the href attribute on the download link.

Links:

Posted by Chad Dotson in Programming, Software Engineering, Technology, Tips, 0 comments

“On The Job Training” vs “On The Job Learning”

A career in programming consists more of “on the job learning” than “on the job training.”

Training

Training as something someone else teaches you to do.  There may or may not be any real expectation of advancing the concepts covered in the training.  The analogy would be that if you work in a sprocket factory, you are not taught about the design of the sprocket nor any concepts behind the design of sprockets, you are taught how to make sprockets given direction.  Training is limited, it can only take you so far in a programming career.

Learning

Learning is progressive and continual and a critical necessity for a career in programming.  You should spend lots of time learning concepts, principles, and techniques.  Then you should attempt apply and further those ideas.  A good programmer is always learning from their experiences and asking, “How can I do this better?,”  “Have I clearly communicated my intentions?,” and “Did I do a good job?”  Will you ever find that you have done a bad job or written crappy code, most certainly!  I think that realization is a sign of progress.

TLDR; Be a good programmer, don’t stop learning.

Posted by Chad Dotson in Key Concepts, Programming, Software Engineering, 0 comments

Installing Technical Analysis Library for Python

I’m tinkering with some financial analysis scripts so when I got to looking into some useful python packages, Technical Anaysis Library popped up.  The python bindings require the TA Lib (Technical Analysis Library) which on osx is available via homebrew.  Now, when I originally installed I didn’t want to install it globally so I’ve got the less preferred, local install setup.  This local install results in the following necessary commands to get the pip package to install correctly.

Now that I’ve brew installed TA Lib and set the new include and library path, I can install the python bindings via pip.

 

Posted by Chad Dotson in Programming, 2 comments

Authorizing a twitter-bot

So last night I ran into a small issue, how to authorize a twitter bot to use an application without stubbing together a website and logging in with the bot account?  The answer, this little script using twython:

Source: https://gist.github.com/moonmilk/035917e668872013c1bd#gistcomment-1398946

Posted by Chad Dotson in Programming, Programming Live Blog, 0 comments

Virtualenvwrapper / Updating Tools

If you are using virtualenvwrapper with the default version of python on your system (OSX in particular) and I recommend updating pip and setuptools as part of your postmkvirtualenv hook.  Just add the following line to YOURVIRTUALENVDIRECTORY/postmkvirtualenv.

 

Posted by Chad Dotson in Programming Live Blog, Tips, 0 comments

Don’t use parse_requirements in your code

I just ran into trouble while building a setuptools package.  Specifically, I was using pip.req.parse_requirements to process package dependencies from a requirements.txt file and noticed that upgrading pip would break my code.  Upon further investigation, I found out that they(authors of pip) do not guarantee the api of their internal code.  This is because they officially do not provide a programmatic api to pip see (https://github.com/pypa/pip/issues/2286).  My suggestion, just use readlines, its the same thing and dirt simple.

Posted by Chad Dotson in Programming, Programming Live Blog, 0 comments