Chad Dotson

A small town Computer Scientist / Software Engineer. Chad enjoys writing Python and JavaScript as well as tinkering with his Raspberry Pi and Arduino. When not programming, he enjoys Photography (especially lightning) and Sci-Fi.

Oct

2020

Compromised

I haven’t posted in a bit and I’ve kind of let the site go a bit. The other day I noticed some questionably named files added to my shared hosting space. Apparently they had been there since June. After trying to figure out if the obscure files were added by the hosting provider during my automatic wordpress updates I’ve decided that I believe at least one of my passwords had been compromised. I have since repaired the altered files that I found and reset passwords. I’m going to more closely monitor the site and make sure it isn’t some plugin misbehaving and ensure the site is not still compromised.

Jan

2020

It’s been awhile

It has been quite a while since I last wrote about anything. It has been hard finding a topic that motivated me enough to write about.

Python 2 EoL

Python 2 has had a long run, longer than it should have. On January 1, 2020, Python 2 reached EoL after almost 20 years of support. Version 2 was originally released in October 2000. I started using it around that time. The first python book I bought was written for version 2.1. When Python 3 was first released in December of 2008, it didn’t go so well. Part of Python’s power is the large number of community packages, however this also proved to slow the adaptation of Python 3 as backwards compatibility was broken. Python 3.5, released in September 2015, was the first time that 3.x shined and about that time the number of packages supporting Python 3 vs Python 2 started to shift. Recently, most popular packages have finally dropped all support for Python 2. I hope you have transitioned to version 3.

More than a destination

Programming is more than a destination. Care about the code you right. Just because code works today doesn’t mean it always will. Complex, bad, or hard-to-understand implementations are a permanent cost.

YAGNI

YAGNI – Your not going to need it. Related to the previous section, adding complexity because you think you will need something is not good. Adding complexity means that you have made change more time consuming, more code to review, understand, change, and test.

Mar

2019

Installing PyAudio on macOS

Installing pyaudio on macOS is almost straightforward, except that you need portaudio installed. The easiest way to remedy this is to use homebrew. If you have your homebrew installing in the recomended location it is as simple as the following:

brew install portaudio
pipenv install pyaudio

1 2	brew install portaudio pipenv install pyaudio

However if you have homebrew using a non-standard location, it requires a few additional settings, if you don’t have them set already.

brew install portaudio
export C_INCLUDE_PATH=~/homebrew/include/:$C_INCLUDE_PATH
export LIBRARY_PATH=~/homebrew/lib/:$LIBRARY_PATH
pipenv install pyaudio

brew install portaudio

export C_INCLUDE_PATH=~/homebrew/include/:$C_INCLUDE_PATH

export LIBRARY_PATH=~/homebrew/lib/:$LIBRARY_PATH

pipenv install pyaudio

Jan

2019

Working with bytes in Python 3

Background

Sometimes you find yourself needing to work at the byte-level in an application you are working on. I feel that in Python there are not enough examples of how to do this. There is also a lot of potential to over-complicate the solution.

This Example

I plan to cover several aspects of working with bytes in this example. I’ll cover working with the struct package, the bytearray built-in and the ctypes module.

The Code

from ctypes import c_int, Structure
from struct import pack_into, unpack_from
  

# create 2 buffers, one smaller than the other for 
# demonstration purposes.
buff1 = bytearray(64)
buff2 = bytearray(32)

# first we'll use the struct package to initalize the arrays.

# initialize buff1 with 32 integers.
pack_into('I' * 16, buff1, 0, *range(0, 16))
print('Buffer 1:', unpack_from('I' * 16, buff1, 0))


# for the sake of demonstration, we'll work with buff2.
# copy part of buff1 into buff2, since we're using
# bytearrays, this should be equivalent to a memcpy
buff2[:] = buff1[:32]

# test it out, did we copy 32 bytes from buff1 into buff2?
print('Buffer 2:', unpack_from('I' * 8, buff2, 0), end='\n\n')

# We can also use the ctypes package to access the buffers

# We can access it piece-meal like this.  Note that this
# copies the buffer, if we didn't want a copy. from_buffer
# is the function we would use.
x = c_int.from_buffer_copy(buff2, 8)
y = c_int.from_buffer_copy(buff2, 12)
print(f'x, y as 2 standalone c_ints: {x.value}, {y.value}', end='\n\n')

# You can also create C Structures to access the data

# Define a simple ctypes structure.
class Point(Structure):
    _fields_ = [
        ('x', c_int),
        ('y', c_int)
    ]

p1 = Point.from_buffer_copy(buff2, 8)
print(f'x, y as elements of a ctype structure (copied): {p1.x}, {p1.y}')

# note that since this is a copy any manipulation doesn't
# effect the buffer.

p1.x, p1.y = 50, 51
print(f'p1.x, p1.y set to: {p1.x}, {p1.y}')
print('Show buff2 is unchanged:', unpack_from('I' * 8, buff2, 0), end='\n\n')

# so, if we wanted to directly manipulate the buffer using the structure
p2 = Point.from_buffer(buff2, 8)
print(f'x, y as elements of a ctype structure (not copied): {p1.x}, {p1.y}')

p2.x, p2.y = 100, 101
print(f'p2.x, p2.y set to: {p2.x}, {p2.y}')

# see that the 3rd and 4th element now been changed.
print('Show buff2 is changed:', unpack_from('I' * 8, buff2, 0), end='\n\n')

# finally note that the original buffer is unchanged.
print('Show buff1 unchanged:', unpack_from('I' * 16, buff1, 0))

from ctypes import c_int, Structure

from struct import pack_into, unpack_from

# create 2 buffers, one smaller than the other for

# demonstration purposes.

buff1 = bytearray(64)

buff2 = bytearray(32)

# first we'll use the struct package to initalize the arrays.

# initialize buff1 with 32 integers.

pack_into('I' * 16, buff1, 0, *range(0, 16))

print('Buffer 1:', unpack_from('I' * 16, buff1, 0))

# for the sake of demonstration, we'll work with buff2.

# copy part of buff1 into buff2, since we're using

# bytearrays, this should be equivalent to a memcpy

buff2[:] = buff1[:32]

# test it out, did we copy 32 bytes from buff1 into buff2?

print('Buffer 2:', unpack_from('I' * 8, buff2, 0), end='\n\n')

# We can also use the ctypes package to access the buffers

# We can access it piece-meal like this. Note that this

# copies the buffer, if we didn't want a copy. from_buffer

# is the function we would use.

x = c_int.from_buffer_copy(buff2, 8)

y = c_int.from_buffer_copy(buff2, 12)

print(f'x, y as 2 standalone c_ints: {x.value}, {y.value}', end='\n\n')

# You can also create C Structures to access the data

# Define a simple ctypes structure.

class Point(Structure):

_fields_ = [

('x', c_int),

('y', c_int)

]

p1 = Point.from_buffer_copy(buff2, 8)

print(f'x, y as elements of a ctype structure (copied): {p1.x}, {p1.y}')

# note that since this is a copy any manipulation doesn't

# effect the buffer.

p1.x, p1.y = 50, 51

print(f'p1.x, p1.y set to: {p1.x}, {p1.y}')

print('Show buff2 is unchanged:', unpack_from('I' * 8, buff2, 0), end='\n\n')

# so, if we wanted to directly manipulate the buffer using the structure

p2 = Point.from_buffer(buff2, 8)

print(f'x, y as elements of a ctype structure (not copied): {p1.x}, {p1.y}')

p2.x, p2.y = 100, 101

print(f'p2.x, p2.y set to: {p2.x}, {p2.y}')

# see that the 3rd and 4th element now been changed.

print('Show buff2 is changed:', unpack_from('I' * 8, buff2, 0), end='\n\n')

# finally note that the original buffer is unchanged.

print('Show buff1 unchanged:', unpack_from('I' * 16, buff1, 0))

Output

Buffer 1: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
Buffer 2: (0, 1, 2, 3, 4, 5, 6, 7)

x, y as 2 standalone c_ints: 2, 3

x, y as elements of a ctype structure (copied): 2, 3
p1.x, p1.y set to: 50, 51
Show buff2 is unchanged: (0, 1, 2, 3, 4, 5, 6, 7)

x, y as elements of a ctype structure (not copied): 50, 51
p2.x, p2.y set to: 100, 101
Show buff2 is changed: (0, 1, 100, 101, 4, 5, 6, 7)

Show buff1 unchanged: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)

Buffer 1: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)

Buffer 2: (0, 1, 2, 3, 4, 5, 6, 7)

x, y as 2 standalone c_ints: 2, 3

x, y as elements of a ctype structure (copied): 2, 3

p1.x, p1.y set to: 50, 51

Show buff2 is unchanged: (0, 1, 2, 3, 4, 5, 6, 7)

x, y as elements of a ctype structure (not copied): 50, 51

p2.x, p2.y set to: 100, 101

Show buff2 is changed: (0, 1, 100, 101, 4, 5, 6, 7)

Show buff1 unchanged: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)

A statement on copying

Be careful with slicing a python bytearray, bytes or array.array. Slicing creates a copy and can impact the performance of your application. There is a better way; enter memoryview. Memoryview works with anything that implements the Python Buffer Protocol and makes slicing very efficient. Slicing a memoryview will result in another memoryview, not a copy of the bytes represented.

Extra Reading

Sep

2018

Python CSV Module Oddity

A Python Oddity

I was using Python to encode a CSV file using a custom dialect recently when I noticed something odd. I noticed that the csv writer class takes an optional argument that enables you to change the line terminator. That’s ok, however, the csv reader class does not honor the argument. So, through the default api, you can create CSV that you cannot read back in with the default api. According to the documentation, this applies to Python 2.7 through 3.7. It also probably applies to versions < 2.7 but that documentation is no longer online.

Python.org Documentation

A Simple Script

I wrote the following simple script to illustrate this oddity. Basically it has a list of lists that it converts to CSV and back using various line terminators, if the output differs from the source, the difference is displayed.

import csv
import io
import re


def convert_to_csv_and_back(rows: list, lineterminator: str):
    with io.StringIO() as o:
        writer = csv.writer(o, lineterminator=lineterminator)
        for row in rows:
            writer.writerow(row)

        with io.StringIO(o.getvalue()) as i:
            reader = csv.reader(i, lineterminator=lineterminator)
       
            return list(reader)

source = [
    ['this', 'is', 'row', '1'],
    ['this', 'is', 'row', '2'],
    ['this', 'is', 'row', '3']
]

lineterminators = ['|', ':', '\t', '\r\n']

for terminator in lineterminators:
    output = convert_to_csv_and_back(source, terminator)
    source_set = set(map(tuple, source))
    output_set = set(map(tuple, output))

    difference = source_set.symmetric_difference(output_set)
    to_from_csv_failed = len(difference)
    
    print(f'Line Terminator: {repr(terminator)}')
    print(f'To/From CSV Worked: {"No" if to_from_csv_failed else "Yes"}')
    print(f'Source: {source}')
    print(f'Output: {output}')
    
    if to_from_csv_failed:
        print(f'Difference: {difference}')

    print('--------------------------')

import csv

import io

import re

def convert_to_csv_and_back(rows: list, lineterminator: str):

with io.StringIO() as o:

writer = csv.writer(o, lineterminator=lineterminator)

for row in rows:

writer.writerow(row)

with io.StringIO(o.getvalue()) as i:

reader = csv.reader(i, lineterminator=lineterminator)

return list(reader)

source = [

['this', 'is', 'row', '1'],

['this', 'is', 'row', '2'],

['this', 'is', 'row', '3']

]

lineterminators = ['|', ':', '\t', '\r\n']

for terminator in lineterminators:

output = convert_to_csv_and_back(source, terminator)

source_set = set(map(tuple, source))

output_set = set(map(tuple, output))

difference = source_set.symmetric_difference(output_set)

to_from_csv_failed = len(difference)

print(f'Line Terminator: {repr(terminator)}')

print(f'To/From CSV Worked: {"No" if to_from_csv_failed else "Yes"}')

print(f'Source: {source}')

print(f'Output: {output}')

if to_from_csv_failed:

print(f'Difference: {difference}')

print('--------------------------')

Results

As you can see from the results here; with the exception of ‘\r\n’, all the various line terminators failed.

Line Terminator: '|'
To/From CSV Worked: No
Source: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]
Output: [['this', 'is', 'row', '1|this', 'is', 'row', '2|this', 'is', 'row', '3|']]
Difference: {('this', 'is', 'row', '1|this', 'is', 'row', '2|this', 'is', 'row', '3|'), ('this', 'is', 'row', '1'), ('this', 'is', 'row', '2'), ('this', 'is', 'row', '3')}
--------------------------
Line Terminator: ':'
To/From CSV Worked: No
Source: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]
Output: [['this', 'is', 'row', '1:this', 'is', 'row', '2:this', 'is', 'row', '3:']]
Difference: {('this', 'is', 'row', '1'), ('this', 'is', 'row', '2'), ('this', 'is', 'row', '1:this', 'is', 'row', '2:this', 'is', 'row', '3:'), ('this', 'is', 'row', '3')}
--------------------------
Line Terminator: '\t'
To/From CSV Worked: No
Source: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]
Output: [['this', 'is', 'row', '1\tthis', 'is', 'row', '2\tthis', 'is', 'row', '3\t']]
Difference: {('this', 'is', 'row', '1'), ('this', 'is', 'row', '1\tthis', 'is', 'row', '2\tthis', 'is', 'row', '3\t'), ('this', 'is', 'row', '3'), ('this', 'is', 'row', '2')}
--------------------------
Line Terminator: '\r\n'
To/From CSV Worked: Yes
Source: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]
Output: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]
--------------------------

Line Terminator: '|'

To/From CSV Worked: No

Source: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]

Output: [['this', 'is', 'row', '1|this', 'is', 'row', '2|this', 'is', 'row', '3|']]

Difference: {('this', 'is', 'row', '1|this', 'is', 'row', '2|this', 'is', 'row', '3|'), ('this', 'is', 'row', '1'), ('this', 'is', 'row', '2'), ('this', 'is', 'row', '3')}

--------------------------

Line Terminator: ':'

To/From CSV Worked: No

Source: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]

Output: [['this', 'is', 'row', '1:this', 'is', 'row', '2:this', 'is', 'row', '3:']]

Difference: {('this', 'is', 'row', '1'), ('this', 'is', 'row', '2'), ('this', 'is', 'row', '1:this', 'is', 'row', '2:this', 'is', 'row', '3:'), ('this', 'is', 'row', '3')}

--------------------------

Line Terminator: '\t'

To/From CSV Worked: No

Source: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]

Output: [['this', 'is', 'row', '1\tthis', 'is', 'row', '2\tthis', 'is', 'row', '3\t']]

Difference: {('this', 'is', 'row', '1'), ('this', 'is', 'row', '1\tthis', 'is', 'row', '2\tthis', 'is', 'row', '3\t'), ('this', 'is', 'row', '3'), ('this', 'is', 'row', '2')}

--------------------------

Line Terminator: '\r\n'

To/From CSV Worked: Yes

Source: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]

Output: [['this', 'is', 'row', '1'], ['this', 'is', 'row', '2'], ['this', 'is', 'row', '3']]

--------------------------

Closing Thoughts

So, why would you want to specify the line terminators with the CSV module? There are probably only a handful of reasons, my only reason would be a custom dialect where I wanted to ensure that there were no carriage returns or line feeds in. 99.9% of the time, you want ‘\r\n’.

Aug

2017

Solar Eclipse 2017 From Sparta, TN

The August 21, 2017 Solar Eclipse as photographed from Sparta, TN (35.9727° N, 85.5638° W). This was about 3.2 miles from the center of Totality! These images do not convey the magnitude of the experience. Knowing what to expect in general does not prepare you. If I tried to find a single word to describe it, I would pick astonishing.

: Chad Dotson – 2017 Solar Eclipse – Sparta, Tennessee (35.9727° N, 85.5638° W) – 2m38.8s of Totality

: Chad Dotson – 2017 Solar Eclipse – Sparta, Tennessee (35.9727° N, 85.5638° W) – 2m38.8s of Totality

Location	Sparta, TN (35.9725° N, 85.5638° W)
Duration of Totality	2m38.8s
Magnitude	1.014
Obscuration	100.00%

Event	Date Time (UT)	Alt	Azi
Start of partial eclipse (C1)	2017/08/21 17:01:17	63.9°	154.8°
Start of total eclipse (C2)	2017/08/21 18:29:50.1	63.9°	205.4°
Maximum eclipse	2017/08/21 18:31:09.6	63.8°	206.1°
End of total eclipse (C3)	2017/08/21 18:32:28.9	63.6°	206.8°
End of partial eclipse (C4)	2017/08/21 19:56:17.0	51.9°	239.1°

Jul

2017

Thoughts On Team Communication

Awhile back I had some thoughts on communication. If you’ve ever played World of Tanks Blitz you’d know that basically its a team of tanks against another team of tanks. With the pick up, fast paced nature communication is minimal at best (sometimes limited to a single “<<<<<<<<<” or “>>>>>>>>>” indicating which direction to take the offense). I found that teams that could coordinate with minimal communication, play their tank roles (scouts, mediums, heavies, and destroyers), and move fast could achieve massive overwhelming victories. Something similar is probably true in an agile/teamwork environment. Know your stuff, know your role, take opportunities, work together, succeed.

Jun

2017

iPhone 5 / Verizon 4G – No Bars / 3g No Service Fix

Problem:

You’ve got a iPhone 5 that reports that it has 4g LTE service but 0 bars and, if you turn off LTE, it reports No Service under 3g.

Solution:

The fix, turn off all roaming . Settings > Cellular > Cellular Data Options > Roaming – Turn off all settings here.

Python 2 EoL

More than a destination

YAGNI

Background

This Example

The Code

Output

A statement on copying

Extra Reading

A Python Oddity

A Simple Script

Results

Closing Thoughts

Recent Posts