SSLError

Have you ran into trouble with SSLErrors after upgrading to Python 3.6 on macOS Sierra?  For me, my first encounter was downloading the bokeh sample data (included here).

I finally got a chance to research exactly what was wrong.  Given that Python 3.5 was working on my machine and 3.6 was working on several linux machines, I thought that it could have been an issue with Python 3.6, unlikely as it may be.  After no bug fix version was released I realized that it must be my configuration.

Searching, I found one result related to this in the Python issue tracker: Issue 28150.  One answerer points out that Python on macOS no longer relies on Apple’s version of OpenSSL, instead it is shipped with a new one.  The gotcha: this new one does not have trust certificates installed.  All this is detailed in the Readme.

The Installer Readme

So, as the Python issue tracker mentioned, there is an entry in the installer readme.  Perhaps I should read these closer…

Certificate verification and OpenSSL

**NEW** This variant of Python 3.6 now includes its own private copy of OpenSSL 1.0.2.  Unlike previous releases, the deprecated Apple-supplied OpenSSL libraries are no longer used.  This also means that the trust certificates in system and user keychains managed by the Keychain Access application and the security command line utility are no longer used as defaults by the Python ssl module.  For 3.6.0, a sample command script is included in /Applications/Python 3.6 to install a curated bundle of default root certificates from the third-party certifi package (https://pypi.python.org/pypi/certifi).  If you choose to use certifi, you should consider subscribing to the project’s email update service to be notified when the certificate bundle is updated.

The bundled pip included with the Python 3.6 installer has its own default certificate store for verifying download connections.

Two Solutions

There are two solutions as mentioned:

  • Install the certifi package.
  • Run the download script provided with the installer – /Applications/Python 3.6/Install Certificates.command.

For myself, I ran the bundled script and everything seems to be functioning fine.  It seems more than a little odd that this was not done on installation.

 

 

Pip installs From github

Installing a package from github is fairly simple.  The following are examples of installing packages from github.

requirements.txt

At some point, you will find yourself wanting to list a dependency in the requirements.txt file that resides on github.  This is fairly straight forward If you only plan to use it in requirements.txt (not processed for usage in setup.py).  Note that I’ve specified the master branch of repo and given it a version id.

setup.py and requirements.txt

Some people will process their requirements.txt files to generate the install_requires parameter for the setup function called in setup.py.  This works fine until you have a repository on github.  Setup will fail to find your dependencies if your requirements.txt has a line like the one above.  To remedy this we must do two things.

  1. Parse the line to create a named python dependency for install requires. Given the file above, install_requires would equal ["repo==0.0.1"] .
  2. Specify the dependency_links argument to setup.  ["https://github.com/chaddotson/repo/tarball/master#egg=repo-0.1.1"]  for this example.

An Example

This is an example of a setup.py that properly processes requirements.txt dependencies that are located on github.  It’s probably not complete, but it works for what I need.  Feel free to take and adapt.  See the repo here (python3).

 

 

 

 

Have you ever encountered a situation where it would be beneficial to save whatever is written to the console across browser sessions?  If you have, you know there are not many solutions.  I recently decided that localStorage would be a good candidate to implement a solution to this problem with.

logsaver.js

logsaver.js is a lightweight, simple log preservation script.  It utilizes the flexibility of JavaScript to replace the default console.log function with a new, customized one.  Along with writing messages to the original console.log, this custom console.log  also writes messages to the specified localStorage key.

  • To start saving console.log messages, call startSavingLog().
  • To stop saving console.log messages, call stopSavingLog().
  • To clear saved console.log messages, call clearSavedLog().
  • To completely remove the localStorage key used by logsaver, call removeSavedLog().

For more information on options and how to use them, checkout the readme.

While you can copy the source below, I strongly recommend grabbing the latest from github here: logsaver.js.

Notes:

  • It doesn’t handle objects well.  Results in [Object object] appearing in the localStorage key value.  This is because localStorage is limited to strings.
  • It is possible to exceed the maximum space allocated to you in localStorage.  For desktop browsers, Chrome and Firefox have a 10 MB limit, while Safari and IE both have a 5 MB limit.  These number differs for mobile browsers.  More details.  In this event, logsaver.js will stop trying to save the log and display additional error messages in the console.

v0.1.0 Source

Have you encountered the following scenario?

You are trying to solve a problem (or helping solve a problem) and know or at least think you know the solution.  You are in the middle of implementing it when someone else looks at it and says, “why don’t you do it this way, isn’t this way easier/better?”  Taking a step back, you realize that the question not only has merit but is a better and much more obvious solution; you can’t believe you missed it.

What happened?

I think its because you were too close to the problem and had developed a very narrow focus.  That narrow focus prevented you from seeing the better solution.  Perhaps this is even a variation of functional fixedness in that we’ve latched onto an idea of how to solve a problem and our mind’s may not see alternatives easily.

What can we do?

  • Think about the broad (or product) level goals regularly.
  • Entertain questions and/or suggestions from others.
  • Ask: “Is this the best way?”
  • Ask: “Is this the practical way?”
  • Don’t overthink the problem.
  • Get it working then evaluate the solution and/or do a code review!

So, you want to create a web worker from a function, the problem is all the examples examples show creating a web worker using a separate JavaScript file or from embedded JavaScript source.  There is a way to do it and that is using Blob URLs.  In the example below, I have taken the simple dedicated worker example from mdn and modified it appropriately.

First, we assume that we have a function that we want to serve as the source of the web worker.

Next, we use that function’s source to create the web worker.

See this example in action on jsfiddle here: http://jsfiddle.net/chaddotson/jr5p3L7r/

Notes:

  • The createObjectURL function is experimental according to MDN, so that is a bit of a warning bell.  However, according to caniuse, it is mostly safe across all browsers.
  • This can be used to create shared web workers, but they will not be shareable between tabs or windows.
  • This makes it more difficult, but possible to assemble dependencies for the web worker.
  • This isn’t the officially supported way to create a web worker.
  • I think this approach to creating a worker should be used sparingly, if ever.

 

Many of us have wanted to create downloadable content straight from the browser.  I remember several years back where it was more difficult and the solutions were not clean.  Today, it is much much more simple and clean.  Enter Blob URLs.

In this example, I have an object of arrays.  These arrays may or may not all be the same length.  The csvify function accounts for this by letting the user optionally specify a default value and whether or not the pad at the beginning or at the end for arrays that aren’t the same size as the max.

It’s what comes after the csvify function that allows all this to work.  To make this csvified data downloadable, I first create a blob.  This blob contains the csv data and is created with the csv mime type.  I then create a url from that blob using createObjectURL and set that url as the href attribute on the download link.

Links:

If you are using virtualenvwrapper with the default version of python on your system (OSX in particular) and I recommend updating pip and setuptools as part of your postmkvirtualenv hook.  Just add the following line to YOURVIRTUALENVDIRECTORY/postmkvirtualenv.

 

Background

Sometimes we may generate or retrieve a list, set or even dict when creating collection of things that we will be testing against.  Theoretically a set, frozenset or dictionary should be the fastest (and equivalent) forms of storage for this operation.  However, what I’ve found is that on some systems the set is faster and on others dict is faster.  This difference maybe very important if your writing real-time or close to real-time software.  So where am I going with this?

Big-O Notation – Advertised Complexity

Python has published the expected time complexities of their collection types.  I’ve copied the ones for the in operator below.  These Big-O numbers are exactly what you would expect since everything but a list is implemented using a hashing algorithm.  It should be noted, however, that the speed of the set, frozenset, and dict can be compromised if the objects stored do not implement a good hashing algorithm.

Type Average Worst
list O(n)
set O(1) O(n)
frozenset O(1) O(n)
dict O(1) O(n)


More: Python Time Complexity

What I Found

Going back to my statement above, I found that on certain machines, python sets were faster and on some machines python dicts where faster.  I cannot replicate sets being faster in all cases directly so I tried to replicate it with a RHEL 7.1 machine on AWS.  Given that I was at an optimal case for the collection (no collisions), I would have thought that set, frozenset, and dict at least performed on par with each other.  I was surprised to find with the default python interpreter my tests showed that python dicts are actually faster.  So, I reran the tests with the corresponding version of PyPy and found that the expected results hold true and set and frozenset operate at virtually the same speed as dicts.  I suspect the primary reasons for the differences are the compiler used to create the python binaries.  It was interesting however that PyPy performed as expected on all systems.

The Data

I ran the benchmarks on OSX, Ubuntu 14.04, and RHEL 7.1 (Courtesy of AWS Free Tier);  Though, I opted not to record the RHEL results as they are similar to the Ubuntu results.

Benchmarks Fastest % Difference
OSX
Python
list 5.47 150.641
set 0.85 9.877
frozenset 0.85 9.877
dict 0.77 0.77 0.000
PyPy
list 0.34 89.362
set 0.13 0.13 0.000
frozenset 0.13 0.13 0.000
dict 0.13 0.13 0.000
Ubuntu
Python
list 6.07 123.733
set 1.44 0.697
frozenset 1.49 4.110
dict 1.43 1.43 0.000
PyPy
list 0.78 102.913
set 0.25 0.25 0.000
frozenset 0.25 0.25 0.000
dict 0.26 3.922

Recommendations

If you have a need to create a collection to test for existence like in this example; favor set, frozenset or dict whichever makes sense for your situation.  If you are working with a list your given and you want to speedup the system, you can consider changing the list to a set.

The Code

I’ve uploaded all the code to github.  It is available here: https://github.com/chaddotson/container-membership-benchmark/.