knanne

Notes On Raspberry Pi

2019-12-07T00:00:00+00:00

A quick compilation of notes on Raspberry Pi

(Headless) Rasbperry Pi Setup
Pi Config
Fix Pi to Static IP
Setup DDNS Dynamic Update Client (DUC)
Setup OpenVPN Client

(Headless) Rasbperry Pi Setup

Download latest version of Raspbian (Lite) and flash it to an SD card. docs on flashing
Create empty ssh file and place it on boot partition. docs on ssh
```
C:\Users\Kain
>>> E:
E:\
>>> touch ssh
```
Create and fill out a wpa_supplicant.conf file and place it on boot partition using the copy (cp) command below. docs on wifi config
```
E:\
>>> cp C:\Users\Kain\Documents\RaspberryPi\wpa_supplicant.conf wpa_supplicant.conf
```
Connect Pi to Router via LAN, and connect power. Pi should boot and flash red power light, and green internet light.
Scan Network to identify IP address of PI, probably corresponding to “Raspberry Pi Foundation”. If on Windows, can use advanced-ip-scanner
SSH into the PI’s IP address using u: pi p: raspberry. If on Windows, can use putty.
Change the default password by running the command passwd. docs on users

Pi Config

run sudo raspi-config to configure the PI. docs on config

Fix Pi to Static IP

Great answers to the question here.

Show IP address of PI with ip -4 addr show | grep global.
Show IP address of router with ip route | grep default | awk '{print $3}'
Edit the config file by running sudo nano /etc/dhcpcd.conf. docs on nano

Enter the following essential information (there is a section in the template for this). One section for ethernet connection, and one for WLAN.

interface eth0
static ip_address=[YOUR PI IP ADDRESS]
static routers=[YOUR ROUTER IP ADDRESS]
static domain_name_servers=[YOUR ROUTER IP ADDRESS]

interface wlan0
static ip_address=[YOUR PI IP ADDRESS]
static routers=[YOUR ROUTER IP ADDRESS]
static domain_name_servers=[YOUR ROUTER IP ADDRESS]

Ctrl+X to exit nano
Type y to save changes when prompted
Hit Enter to confirm file for writing and complete exit
Reboot the Pi with sudo reboot

Setup DDNS Dynamic Update Client (DUC)

Keep your current IP address in sync with your No-IP host or domain with our Dynamic Update Client (DUC). - no-ip

Full instructions for no-ip service here: https://www.noip.com/support/knowledgebase/install-ip-duc-onto-raspberry-pi/

Setup OpenVPN Client

Using PiVPN

SSH into your pi and install pivpn with curl -L https://install.pivpn.io | bash
You can confirm default settings for most options. When prompted, enter your own domain name. You can get a free one from noip
Check that the service is running sudo service openvpn status
Create a new client certificate by running pivpn add and giving it a password
Copy the certificate file from pi to the device you want to connect from. If on Windows, can use filezilla
Next, configure your DDNS service on your router. If you are using a two router setup (modem/router), do this on the router acting as a modem where the internet comes in first.
Lastly, forward the OpenVPN service port on your router, the default port from piVPN is 1194. Important note here, OpenVPN uses UDP as the protocol. If you are using a two router setup (modem/router) you may need to first forward traffic on that port to the second router, to then be forwarded to the pi.

Useful PiVPN commands:

List certificates with pivpn list
Revoke a compromised certificate with pivpn revoke
Add a new certificate with pivpn add optionally with nopass for no password
List connected clients with pivpn -c
Reboot the openvpn server with sudo service openvpn reboot
Note that the VPN server automatically starts after a reboot, so nothing is needed if power to the pi cuts out temporarily

Notes On Excel

2018-05-29T00:00:00+00:00

Random notes on using Excel

Pivot Tables
Filtering Data
Highlight Non-NULL Cells

I know, I know…but sometimes it’s just easier.

Here are some basic things which are handy to know in Excel.

Pivot Tables

Select the rows / columns you want, and go to Insert > Pivot Table. Then logically put the columns in appropriate data categories. For a more detailed description see this tutorial by Microsoft.

Filtering Data

Filter all columns in a sheet? Select the first cell with data and go to Data > Filter. Seriously, it’s that easy.

Highlight Non-NULL Cells

Example of how to highlight non-blank cells in column A.

Go to Home > Conditional Formatting > New Rule.
Select “Use a formula to determine which cells to format”`
Enter the formula =NOT(ISBLANK($A1))
Configure your desired formatting (i.e. fill yellow)
Click Okay
Change “Applies to” to your column =$A:$A
Click Apply

Notes On Shell

2018-05-26T00:00:00+00:00

Random notes on the Unix command line (e.g. bash)

Intro
Advantages
Commands
Scripting

Intro

The following is a compilation of commands and notes on using the command line. Specifically bash. Bash commands, or the corresponding ELF executables, are available on Linux/Unix based systems like Ubuntu and Mac OSX.

While they are not easily available on Windows, it is possible to run some commands via console emulators like Cmder with Git Bash, or more recently the Windows Subsystem for Linux which gives full access to ELF binaries.

For an introduction on the command line, there is a great practical book by Jeroen Janssens called Data Science at the Command Line, which can be read online here.

Generally speaking, the notes provided here are practical data science examples on the collection of tools exposed via the Unix command line (i.e. bash, shell, console). This can include the defaults or commonly used extensions.

Advantages

Unix commands execute specific scripts or tools which have been designed to do a single task, and do it as efficiently as possible without loading all the data in memory. In this case, if available, it is much faster to read/manipulate/process data using these tools than trying to do the same thing in another programming language.

Commands

Below is just a short list of notable commands I have found useful, and want to remember. On all commands, use --help flag for more info and usage instructions.

echo writes to output
printf prints with formatting
touch creates an empty file (this command will NOT overwrite or delete contents of an existing file)
> also known as redirect, outputs text to a file, creating it if it does not exist. > will overwrite an existing file, while >> will append to an existing file
head preview first 10 lines of file
sort sort a file
grep search a file for a regular expression pattern.
| also known as pipe operator, combines multiple commands
find will find a file
cat will concatenate files or output
truncate with -s 0 option will delete contents of a file
uniq will deduplicate data, however the duplicates must be adjacent (use sort first). Use flag -c to count the number of duplications. Also pass -d flag to show duplicated data only.
wc command on a file by default will return the number of lines, words, and characters.

Text Editors

Nano

nano file.txt

Save a file with Ctrl+o, exit a file with Ctrl+x

For Loops

A one line for loop looks like for i in {1..10}; do echo "Hello World!"; done

Examples

Basic File Manipulation

The is an example of how to use practical commands in sequence to explore file operations. Run the one by one in order.

echo Hello World!

touch data.txt

for i in {1..10}
do
  echo "$i Hello World!" >> data.txt
done

head data.txt

sort data.txt -r

grep [3-7] data.txt

for i in {1..15}; do echo "$i Hello World!" >> data.txt; done

wc data.txt

sort data.txt | uniq -c

Curl

curl is used in command lines or scripts to transfer data. - website

jq

jq is a lightweight and flexible command-line JSON processor. - website

You may need to download it here

Here is an example of how to get the html key from a returned JSON object, using the open Instagram oembed endpoint for one of my posts.

curl -s "https://api.instagram.com/oembed/?url=http://instagr.am/p/BjPSQKJFUYa" | jq -r ".html"

Scripting

.sh shell scripts can be used for absolutely anything.

Example curl API, redirect JSON key to file

This below is an example of writing a shell script to automatically and dynamically redirect API data to a file in one command.

Fetch Latest Instagram Post Embed HTML

I recently created a very simple shell script to fulfill a very simple task for this website.

My problem was I wanted to embed my latest Instagram post on the about page, yet Instagram does not allow to get this dynamically. In this case I have to manually update the embed html for the latest post of interest. This is a hassle, and like any good developer I considered how to automate this.

After finding that Instagram has an open API for getting a posts embed html, it is possible to simply make a GET request, and parse the HTML from the returned JSON. So I explored using curl to do the HTTP request, and jq to process the JSON, and finally redirect this into a file. The following command works well.

curl -s "https://api.instagram.com/oembed/?url=http://instagr.am/p/BjPSQKJFUYa" | jq -r ".html"

Now, before I continue I must admit at this point I found someone who implemented this exact thing and already wrapped it up into a shell script. So credit goes to Wouter Bulten for the informational post on this topic already. My simplified and custom solution meeting my specific needs is continued below.

Now to just finalize the script, I add two dynamic variables. One for the path of the output file, hardcoded in the script. And the other is the required shortcode of the post of interest. The shortcode can be supplied as a command line argument or entered during execution when prompted.

I saved the file under the name, fetch_latest_instagram.sh. Running the code therefore looks like fetch_latest_instagram.sh BjPSQKJFUYa, and then for me the embed code is written to the correct file in my _includes folder which I pick up on my about page.

Note that I added an additional prompt before closing because on Windows the shell script opens in a new window, which automatically closes after execution, making it impossible to read if there was an error.

if [ $# -eq 0 ]
  then
    echo "Enter post shortcode:"
    read shortcode
else
  shortcode="$1"
fi

path="_includes\integrations\instagram_latest_photo.html"

echo "Fetching embed HTML of given post"
echo "Writing to $path"

curl -s "https://api.instagram.com/oembed/?url=http://instagr.am/p/$shortcode" | jq -r ".html" > "$path"

echo "Successfully wrote embed code to instagram_latest_photo.html"

echo "Type any key to close:"
read dummmy

echo "Closing"

Notes On Ssh

2018-05-26T00:00:00+00:00

Random notes on using SSH

SSH
Keys
Connecting
Copy
Execute Command
Execute Local File
Code Snippets
Example of Python Workflow
SFTP

SSH

ssh (SSH client) is a program for logging into a remote machine and for executing commands on a remote machine. - website

Keys

Most implementations of SSH connection will use the secure route of authentication through keys. Meaning authentication is done by comparing the transmission of an encrypted string across hosts. A private-public key pair is first generated on the local machine wishing to connect to the remote. The public key is then shared with the remote machine you wish to connect with, while the private key is never shared. Encryption is done on the remote machine with the public key, and decryption is done with the private key living only on the connecting machine before sending the decrypted message back for verification.

Typically when doing this procedure for services like GitHub or GitLab, it is best to simply follow there docs to ensure your profile is configured correctly to their needs.

SSH keys are typically stored in the root or user’s directory in a folder called ~/.ssh/.

To quickly generate a key pair to be used for a specific task, do the following. Maybe you need to generate it for two remote servers needing to connect, or another user.

ssh-keygen -f <KEYFILE> -C <USERNAME>

Change Password

In case you created a keyfile with a password, and need to change or remove it, use the following command. To remove the password just hit enter when prompted for the new one.

ssh-keygen -p -f <KEYFILE>

Connecting

ssh -i <KEYFILE> <USERNAME>@<REMOTEHOST>

Copy

Execute Command

Execute Local File

Code Snippets

Example of Python Workflow

Move Requirements Over

scp ssh -i <KEYFILE> path\to\local\requirements.txt <USERNAME>@<REMOTEHOST>:\path\to\remote\requirements.txt

Install Requirements

ssh -i <KEYFILE> <USERNAME>@<REMOTEHOST "pip install -r requirements.txt"

Execute Local Python File

cat path\to\local\file.py | ssh -i <KEYFILE> <USERNAME>@<REMOTEHOST> python - > path\to\local\log.txt

SFTP

Secure File Transfer Protocol over SSH. DigitalOcean has a good post to start with.

Here are the necessary basics for connecting and transferring files.

sftp username@remote_hostname_or_IP

get remoteFile localFile

put localFile

Notes On Pyspark

2017-12-27T00:00:00+00:00

Random notes, links, commands or snippets of code related to big data analysis using PySpark (on Databricks).

Intro
Common Techniques
Code Snippets
Machine Learning

Intro

Apache Spark™ is a fast and general engine for large-scale data processing. - website

Spark is made accessible to Python users via the Python API, which is virtually as up to date as the Scala and Java API.

I was introduced to Spark via way of Databricks (Also Apache) cloud platform through my company. Consider it a commercial version of Jupyter or Zeppelin notebooks, language-agnostic, integrated on top of a Spark with a bunch of fancy runtime features.

Going through the registration for a free trial of Databricks and deploying it on a free trial of Amazon AWS takes minutes, and I would highly recommend it as a starting point to getting introduced to Spark. Databricks also has a community edition for learning purposes worth looking into.

Browse the Databricks training material and public notebooks for more.

Regarding Python, the Python API docs are available from Apache and the best source for up to date examples. As well, Databricks has some good notes, but not complete as they are not trying to replace the official docs.

Common Techniques

There are two common ways to select on DataFrames through the Python API of Spark. Either df.select("myCol") or df.selectExp("myCol").

For me .select() is the most intuitive coming from Pandas, however I also always do from pyspark.sql import functions as F and use df.select(F.col("myCol").key) or df.select(F.explode("myCol")).

While others prefer to use .selectExp() which accepts SQL but still returns the DataFrame, and do df.selectExp("myCol.key") or df.selectExp("explode(myCol)").

Group by and aggregation look like this, df.groupby("myCol1", "myCol2").agg(F.countDistinct("myCol3")).

The below table (stolen from a Databricks talk I attended) outlines main SQL functions and their Python API equivalent. The cool thing is these apply the same across Spark APIs (Scala, R, Java etc.)

SQL	DataFame API	DataFrame example (with String column names)
SELECT	select, selectExpr	myDataFrame.select(“someColumn”)
WHERE	filter, where	myDataFrame.filter(“someColumn > 10”)
GROUP BY	groupBy	myDataFrame.groupBy(“someColumn”)
ORDER BY	orderBy	myDataFrame.orderBy(“column”)
JOIN	join	myDataFrame.join(otherDataFrame, “innerEquiJoinColumn”)
UNION	union	myDataFrame.union(otherDataFrame)

Chaining functions together works really well, for example: df.filter(...).select(...).join(...).groupby(...).agg(...).

Also, when dealing with Spark Datasets (RDDs or HIVE tables), nested dictionaries and arrays can be utilized quite powerfully, and accessed in natural Pythonic ways. For example, as show above, access dictionaries using myDictCol.key, and index arrays simply with myArrayCol[index].

Common functions to remember are .withColumn() to add calculated fields to DataFrames or chain more than one explode together, and also .withColumnRenamed() to quickly rename that function-applied column.

After from pyspark.sql import functions as F, you have access to a lot of basic tools, which can be combined in all sorts of ways to analysis. Consider the example of making SQL-accepted date column by combining a string year and month column. df.withColumn(F.to_date(F.concat_ws('-', df.year, df.month, F.lit('01')), format='yyyy-MM-dd').alias("date")).

Code Snippets

Below is a collection of code samples which I have created and needed to use in my work more than once. They may be helpful to others.

Explode with Index

Explode an elements in an array, or a key in an array of nested dictionaries with an index value, to capture the sequence.

The below code creates a PySpark user defined function which implements enumerate on a list and returns a dictionary with {index:value} as integer and string respectively. I apply this to a dummy column “myNestedDict” which contains a key “myNestedCol” to show that this can work on dictionaries as well as arrays.

from pyspark.sql.functions import udf, col, explode
from pyspark.sql.types import ArrayType, MapType, StringType, IntegerType

explodeWithIndex = udf(lambda x: i:str(v) for i,v in enumerate(x) if x is not None, MapType(IntegerType(),StringType()))

df2 = df.filter(col("myNestedDict").isNotNull()).select("id", explode(explodeWithIndex("myNestedDict").myNestedCol))

Reverse Explode

Convert a column of strings to comma separated values list as a single string within a group by statement.

The below sample code will group by a dummy id, and roll up a column of string values into a single column of comma separated values. This could also be considered as a reverse explode.

from pyspark.sql import functions as F

df2 = df.groupby("id").agg(F.concat_ws(",", F.collect_list("myValues"))).withColumnRenamed("concat_ws(,, collect_list(myValues))", "myCSV")

Machine Learning

I have a really basic example of how to fully train and evaluate a Linear Regression Model using a Spark DataFrame, you can find it as a notebook at notebooks/pyspark_linear_regression.

Random Resources For Data Analysis

2017-11-22T00:00:00+00:00

Random notes, links, commands or snippets of code related to data analysis and the like.

GitHub File

GitHub File

Download a single file from GitHub using the curl and a hidden, direct path. The link is https://raw.githubusercontent.com/username/project/branch/file

curl https://raw.githubusercontent.com/username/project/branch/file > outfile

Preprocessing Data In Pandas

2017-11-11T00:00:00+00:00

Notes on preparing a Pandas DataFrame for statistical modeling

Pandas
scikit-learn

Pandas

Scale

Transform all data columns to having range from 0 to 1.

scale = lambda x: x / x.max()
df_scale = df.apply(scale)
df_scale.describe()

Normalize

Transform all data columns to having mean 0, and standard deviation 1.

normalize = lambda x: (x - x.mean()) / x.std()
df_norm = df.apply(normalize)
df_norm.describe()

Standardize

First normalize the data, then scale from 0 to 1.

df_stdz = df.apply(normalize).apply(scale)
# if the above throws an error, this means you probably divided by zero when normalizing. Use the below code to first impute bad data to 0 before scaling.
#df_stdz = df.apply(normalize).replace([-np.inf,np.inf], np.nan).fillna(0).apply(scale)
df_stdz.describe()

scikit-learn

Using scikit-learn, do the same thing.

from sklearnsklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

df_stdz = pd.DataFrame(scaler.fit_transform(df), columns=df.columns, index=df.index)
df_stdz.describe()

Now look into other built-in preprocessors on scikit-learn, and consider how to handle outliers

Notes On Regular Expression

2017-10-04T00:00:00+00:00

Notes and scripts on Regular Expression that may be useful to others

Intro
Basics
Validation
Search
Python
Find and Replace
Random Code Snippets

Intro

Regular Expression is some serious magic.

But, after some time playing around at regex101.com/, you can understand how to combine the right components you may need to solve your problems.

Basics

Here is a short list of some common regex code and their meanings.

^ means start at beginning of sting
$ means start from end of string
anything in [] will return exact matches for those characters
adding a ^ inside the square brackets negates those characters, and returns anything but the given characters, for example: [^0-9] will return non-numeric characters
anything in () means capturing group
a non-capturing group is identified by (?:+whatever+)
the combination of (.*?) will return virtually everything (except for new line)
chain logic together with logical OR using pipe operator |
{4} following a pattern means with length of 4
\b will match a “word boundary”
(?=) is used as a “positive look ahead”, see this simple explanation
(?<=) is used as a “positive look behind”, see this simple explanation

Validation

A good use for regular expression is for validation strings. For example, if you manage some user interface where you want to accept input based on some set criteria.

Comma Separated Words

The following will validate only a list of comma separated characters (lowercase, or uppercase).

^([a-zA-Z ])+(,[a-zA-Z ]+)*$

Comma Separated Numbers

The following will validate a only list of comma separated numbers of length 1 to 10.

^([0-9]{1,10})+(,[0-9]{1,10}+)*$

Email

If we consider an email con only contain characters 0-9, a-z, A-Z, -, ., or _, we can construct checks for these characters separated by @.

Verify string is email with ^([0-9a-zA-Z._-]+)@(?:[0-9a-zA-Z._-]*)$.

Verify string is a list of emails, separated by comma or semicolon, with ^(([0-9a-zA-Z._-]+)@(?:[0-9a-zA-Z._-]*)(\]?)(\s*(;|,)\s*|\s*$))*$.

To achieve both of these and also check for a specific email domain, for example a company email address, use ^([0-9a-zA-Z._-]+)@company.com$

Or for a comma / semicolon separated list ^(([0-9a-zA-Z._-]+)@company.com(\]?)(\s*(;|,)\s*|\s*$))*$

Date

Validate a date in YYYY-MM-DD format. Credit to StackOverflow

^\d{4}\-(0[1-9]|1[012])\-(0[1-9]|[12][0-9]|3[01])$

Search

Regular expression can have many applications in data analysis for searching text data.

Between

Just use prefix(.*?)suffix.

Or combine the above logic with non-capturing groups for more dynamic searching. For example, (?:prefix1|prefix2|prefix3)(.*)(?:suffix1|suffix2|suffix3)

Imagine if you needed to parse contracts or some sort of somewhat standardized document to find optional data fields. You could construct a regular expression to include the possible words that would always appear before and after the desired text (e.g. “party 1 paid…to party 2”).

Python

Here is a, not very practical example, of how to use regular expression in python.

The example text:

The quick brown fox jumps over the lazy dog

import re

s = "The quick brown fox jumps over the lazy dog."

pattern = "(?:slow|quick)(.*)(?:walks|jumps|crawls)(.*)(?:lazy|happy|sleeping)(.*)"

r = re.findall(pattern, s)

print(r)

This will print out: [(' brown fox ', ' over the ', ' dog.')]

Find and Replace

Another common problem to solve may be finding and replacing certain text, using more complicated logic than the traditional method.

Replace Leading or Trailing Characters

Instead of doing a find and replace of an extract string, regex allows you create a logical search for certain characters based on their position in relation to other characters.

For example, imagine the case of have the string "$ 10,000.00 USD", and we want to remove the leading or trailing non-numeric characters and keep only the numeric characters, as this is the valid data we want to store.

Use the positive look ahead/behind concepts in regex to apply the following logic.

^[^0-9 ]+(?=[0-9,\.]) will match leading non-numeric characters preceding a pattern of numeric characters which may also include comma or period

(?<=[0-9,\.])[^0-9]+$ will similarly match the trailing non-numeric following a pattern of numeric characters which may also include comma or period

Wrapping this up in Python, with the re.sub() function looks like:

s = "$ 10,000.00 USD"
print(s)
s = re.sub(r'^[^0-9]+(?=[0-9,\.])','',s)
print(s)
s = re.sub(r'(?<=[0-9,\.])[^0-9]+$','',s)
print(s)

Random Code Snippets

Clean Date Text

Here is a simple implementation of cleaning date text, from this StackOverflow post and answer

Imagine parsing text with a date that looks like: “24th of April, 2018”. Wish there was a way to identify these number endings to remove them? Enter (?<=\d)(st|nd|rd|th)\b.

By adding more logic or chaining other cleaning methods in python, you could try and capture many once dirty date text chunks into datetimes types.

import pandas as pd
import re

s = "24th of April, 2018"
s = re.sub(r"(?<=\d)(st|nd|rd|th)\b", '', s)
s = ' '.join(s.replace(',','').replace('of','').split())
d = pd.to_datetime(s, infer_datetime_format=True)
d

This will result in:

Timestamp('2018-04-24 00:00:00')`

Cleaning Data In Pandas

2017-09-24T00:00:00+00:00

Helpful scripts for data “munging” data using Pandas

Columns
Missing Data
Coalesce
Dedupe
Search
Explode

Columns

I often wish to standardize the header of a dataset after import, because the creator used capital letters, spaces, or other non-alphanumeric characters when naming columns.

import re
clean = lambda s: re.sub('[^0-9a-zA-Z]+', '_', s.lower())
df.columns = df.columns.map(clean)

Missing Data

Count how many non-NULL values are in each column.

df.count()

A fast and effective way to count how many NULL values are in each column.

df.isnull().sum()

Therefore, to get a percent of non-NULL values in your dataframe, do:

df.count() / (df.count() + df.isnull().sum())

Impute all NULL values in an entire DataFrame of numeric data to 0.

df.fillna(0)

If you need to impute more than NULL values (maybe because you divided by 0 and now have infinite), you can chain together a .replace() and .fillna().

df.col.replace([-np.inf, np.inf], np.nan).fillna(0)

Coalesce

The concept of applying coalesce to multiple columns (like in SQL) can be applied on DataFrame columns. For example if you wanted to set a master phone number for record based on priority of phone1-3, you could do something like the following.

df.iloc[:,'phone'] = np.nan
df['phone'] = df['phone'].fillna(df.phone1).fillna(df.phone2).fillna(df.phone3)

Dedupe

Dropping duplicates using df.drop_duplicates() is the simplest method of deduplicating records by far. You can give keyword arguments to make it more useful like a only deduplicating on a subset of columns, or a method for which row to take.

If you need to show the duplicates, you can use “~” to negate the indices of the deduplicated rows.

df.loc[~df.isin(df.drop_duplicates())]

To show all rows within duplicated records based on some subset of columns, use the following. This can help isolate duplicated data to then figure out the logic for keeping the desired record.

df[df.duplicated(subset=['col1','col2','col3'],keep=False)].sort_values(by='col4',ascending=False)

Instead of keeping the first or last, we can apply more complex logic to for keeping records. For example, keep the record with max value (in col4) for each combination of multiple columns (col1, col2, and col3). Achieve this with the following code, by slicing the your DataFrame on indices of max for each group.

df.iloc[df.groupby(['col1', 'col2', 'col3']).col4.idxmax()]

Search

Search a column by a regular expression pattern, or single keyword, with .str.contains() or .str.match(). For more on this, refer to the Pandas docs on working with text data. Alternatively, filter on a list of keywords with the below code.

df = pd.DataFrame({'text' : ['somekey1blah',
                             'somekey2blah',
                             'somekey3blah',
                             'somekey4blah',
                             'somekey5blah']})

keywords = ['key1', 'key2', 'key3']

# inclusive search
df[df['text'].str.contains('|'.join(keywords), case=False)]
# exclusive search
df[df['text'].str.contains('|'.join(keywords), case=False) == False]

Explode

df = pd.DataFrame({'csv' : ['value1',
                             'value1,value2',
                             'value1,value2,value3,value4,value5,value6',
                             'value1',
                             'value1,value2,value3,value14']})

df_exploded = df['csv'].str.split(pat=',', expand=True).stack().to_frame()

df_exploded.columns = ['value']
df_exploded.index.names = ['id', 'sequence']

df_exploded = df_exploded.reset_index(level='sequence')

df = df.merge(df_exploded, how='left', left_index=True, right_index=True)

Automating Everything In Python

2017-09-03T00:00:00+00:00

Helpful scripts for automating everything Python

Files
Shell
Security
System
HTTP
SQL
Processing
Runtime
SMTP Email
Jupyter

Files

Manipulate files for automating ETL pipelines.

Stats

import os
from datetime import datetime
f = 'file.txt'
size = os.stat(f).st_size
last_modified = datetime.fromtimestamp(os.stat(f).st_mtime)

Temporary

from tempfile import TemporaryDirectory
with TemporaryDirectory() as tmpdir:
    # do stuff
    # context manager cleans up after itself
# OR
import tempfile
import shutil
tmpdir = tempfile.mkdtemp()
# do stuff
# and clean up after yourself
shutil.rmtree(tmpdir)

Creating

import os
folder = os.path.abspath('folder')
if not os.path.exists(folder):
    os.makedirs(folder)

Copying

import os
import glob
import shutil
src_dir = 'folder_in'
dst_dir = 'folder_out'
if not os.path.exists(dst_dir):
  os.makedirs(dst_dir)
for f in glob.glob(os.path.join(src_dir, '*.txt')):
  shutil.copy(f, dst_dir)

Zipping

import zipfile
import os
src_dir = 'folder'
dst_dir = 'folder.zip'
zp = zipfile.ZipFile(dst_dir, 'w')
for folder, subfolder, files in os.walk(src_dir):
  for f in files:
    zp.write(os.path.join(folder,f), os.path.relpath(os.path.join(folder,f), dst_dir), compress_type=zipfile.ZIP_DEFLATED)
zp.close()

Decompressing

import gzip
import io
infile = 'compressed.gz'
outfile = 'decompressed.txt'
with gzip.open(infile, 'rb') as inf:
  with io.open(outfile, 'wb+') as outf:
    outf.write(inf.read())

Shell

Run command line operations from within python.

Executing Commands

import subprocess
infiles = os.path.join('folder', '*.gz')
outfile = 'combined.gz'

# windows
cmd = 'copy /B /Y {} {}'.format(infiles, outfile)
# unix
cmd = 'cat {} > {}'.format(infiles, outfile)

subprocess.run(cmd, shell=True)

Security

Always hide credentials or keys behind the scenes.

Accessing Credentials

From files. The file should contain a dictionary, looking something like this:

{
  'username': 'fake_user',
  'password': 'fake_password'
}

import json
with open('credentials', 'r') as f:
    creds = json.loads(f.read())

From environmental variables.

import os
username = os.environ['USERNAME']
password = os.environ['PASSWORD']

System

Access and use system settings in a workflow.

Time

import time
from datetime import datetime

start = datetime.now()
# do something
time.sleep(2)
# or sleep two seconds
end = datetime.now()

time_delta = end - start
processing_speed = time_delta.seconds

OS

import platform
print(platform.system())

Python Installation

import sys
# python installation location
print(sys.executable)
# python version
print(sys.version_info.major)

Modify PATH

Temporarily add a folder to you path, maybe for importing a custom python module.

import sys
import os
custom_module = os.path.abspath('custom_module')
sys.path.append(custom_module)

HTTP

Operations over HTTP(s)

API

import requests
url = 'https://site/endpoint'
payload = {
  'item1': item1,
  'item2': item2
}
headers = {'hitem1': hitem1}
r = requests.post(url, headers=headers, json=payload)

Download

import urllib
url = 'https://site/endpoint/file'
urllib.urlretrieve(url)

SQL

Connect to a database and execute a sql statement with the following.

This following example uses SQLAlchemy. Depending on the type of database you are connecting to, there is an extra configuration step to install the right Python-to-ODBC connection, then choose the correct engine below.

PostgreSQL: pip install psycopg2, see psycopg homepage

MySQL: pip install mysqlclient, see mysqlclient-python on github which supports python 3.6. If on Windows, find the correct wheel file for your platform on pypi

Oracle: pip install cx_Oracle see python-cx_Oracle on github. You will also need the Oracle Client.

import json
# read database credentials from json file
with open('creds_file') as f:
  creds = json.loads(f.read())

from sqlalchemy import create_engine
# create appropriate database connection engine

#postgresql
db = create_engine('postgresql+psycopg2://{}:{}@{}:{}/{}'.format(
  creds['user'],
  creds['password'],
  creds['host'],
  creds['port'],
  creds['database']),
    encoding='utf8')

#mysql
db = create_engine('mysql+mysqldb://{}:{}@{}:{}/{}?charset=utf8&local_infile=1'.format(
    creds['user'],
    creds['password'],
    creds['host'],
    creds['port'],
    creds['database']),
      encoding='utf8')

#oracle
db = create_engine('oracle+cx_oracle://{}:{}@{}:{}/{}'.format(
  creds['user'],
  creds['password'],
  creds['host'],
  creds['port'],
  creds['tns']),
    encoding='utf8')

#write some data to file from pandas
#(format NULLs for MySQL acceptance)
df.fillna('\\N').to_csv('datafile.txt', sep='\t', encoding='utf8', quotechar='"', line_terminator='\n')

# define some sql statement
sql = """
  LOAD DATA LOCAL INFILE '{data}'
  INTO TABLE {schema}.{table}
  CHARACTER  SET utf8
  FIELDS TERMINATED BY '\t'
  OPTIONALLY] ENCLOSED BY '"'
  LINES TERMINATED BY '\n'
  IGNORE 1 LINES
""".format(data='datafile.txt',
           schema='db',
           table='data')

# execute sql
connection = db.connect()
r = connection.execute(sql)
connection.close()

Processing

Progress

tqdm is a super nice library for showing a progress bar when executing loops. They even have a pandas integration.

It runs in terminal by default, you can also run it in a Jupyter Notebook. Here is good example, using enumerate.

from tqdm import tqdm
from tqdm import tnrange, tqdm_notebook

with tqdm_notebook(total=len(data)) as pbar:
  for i,file in enumerate(data):
    pbar.update(1)

Runtime

Various procedures to consider when building .py executables.

Arguments

Pass command line arguments to a .py during execution. Simply copy the below to a test.py file and run test.py -h from the command line.

import argparse
parser = argparse.ArgumentParser(description='Example on how to use ArgumentParser', formatter_class=argparse.ArgumentDefaultsHelpFormatter)

def bool_string_input(s):
    if s in ['True','False']:
        return s == 'True'
    else:
        raise ValueError('Incorrect boolean value provided. Please specify one of "True" or "False"')

parser.add_argument('-a', '--argument', dest='myargument', type=str, help='Example argument description.', default='test')
parser.add_argument('-b', '--boolean', dest='myboolean', type=bool_string_input, help='Example boolean argument description.', default=False)

args = parser.parse_args()

myargument = args.myargument
myboolean = args.myboolean

Logging

Setup a basic logging configuration for outputting log messages to console during runtime.

import Logging

logger = logging.getLogger('MyScript')

ch = logging.StreamHandler()
ch.setLevel(logging.INFO)

formatter = logging.Formatter('%(asctime)-15s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)

logger.addHandler(ch)

logger.info('Doing stuff')
logger.info('Doing other stuff')

SMTP Email

Find a full working example of how to send emails with attachments, in python, using the SMTP protocol.

import smtplib
from email.message import EmailMessage
from email.mime.application import MIMEApplication
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

message_text = """
Automated message sent form Python job.

{custom}
""".format(custom="some custom message")

message_html = """
<html>
  <head></head>
  <body>
      <p>Automated message sent form Python job.</p>
      <br><br>
      <p>{custom}</p>
  </body>
</html>
""".format(custom="some custom message")

msg = MIMEMultipart('alternative')

msg_text_part = MIMEText(message_text, 'plain')
msg_html_part = MIMEText(message_html, 'html')

msg.attach(msg_text_part)
msg.attach(msg_html_part)

msg['Subject'] = 'Python Job Finished'
from_email = 'your_email@domain.com'
msg['From'] = from_email
to_email = 'their_email@domain.com'
msg['To'] = to_email

for f in attachments:
    with open(f,'rb') as b:
        part = MIMEApplication(
            b.read(),
            Name=os.path.basename(f)
        )
        part['Content-Disposition'] = 'attachment; filename="{}"'.format(os.path.basename(f))
        msg.attach(part)

s = smtplib.SMTP('smtp-server.yourcompany')
s.sendmail(from_email, to_email, msg.as_string())

Jupyter

Jupyter Notebooks are a great place to write and debug code on the fly and interactively. Here are a few added magic functions or tricks to make going back and forth a bit more convenient.

Run a python file from within a Jupyter Notebook with:

!python program.py -h

Load a python file into a Jupyter cell with the following.

%load program.py

And if running some code inside a Jupyter Notebook, which utilizes argparse and thus undesirable breaks execution, you can allow Jupyter Notebook to pass this section by doing the following.

if get_ipython().__class__.__name__ == 'ZMQInteractiveShell'
    sys.argv = ['-f']