Notify with Python
Working in Python, I often run data processing, transfer, and model training scripts. Now, with any reasonable degree of complexity and/or big data, this can take some time.
Although often enough we all have some other work to be done whilst waiting for these processing to complete, occasionally, we don’t.
For this purpose, I put together a set of Python scripts built for this exact problem. I use these scripts to send process updates, visualizations and completion notifications to my phone.
So, when we do occasionally have those moments of freedom. We can enjoy them without being worried about model progress.
What we need
Okay so the first thing we need to ask is — what do we need to know?
Now of-course, this really depends on the work you are doing. For me I have three main processing tasks that have the potential to take up time:
- Model training
- Data processing and/or transfer
- Financial modelling
For each of these, there are of-course different pieces of information that we need to stay informed about. Let’s take a look at an example of each.
Model Training
Update every n epochs, must include key metrics. For example, loss and accuracy for training and validation sets.
Notification of completion (of-course). For this I like to include:
- prediction outputs, for text generation, the generated text (or a sample of it) — for image generation, a (hopefully) cool visualization.
- visualization of key metrics during training (again, loss and accuracy for both training and validation sets)
- other, less essential but still useful information such as local model directories, training time, model architecture etc
Let’s take the example of training a neural network to reproduce a given artistic style.
For this, we want to see; generated images from the model, loss and accuracy plots, and current training time, and a model name.
import notify
START = datetime.now() # this line would be placed before model training begins
MODELNAME = "Synthwave GAN" # giving us our model name
NOTIFY = 100 # so we send an update notification every 100 epochs
# for each epoch e, we would include the following code
if e % notify_epoch == 0 and e != 0:
# here we create the email body message
txt = (f"{MODELNAME} update as of "
f"{datetime.now().strftime('%H:%M:%S')}.")
# we build the MIME message object with notify.message
msg = notify.message(
subject='Synthwave GAN',
text=txt,
img=[
f'../visuals/{MODELNAME}/epoch_{e}_loss.png',
f'../visuals/{MODELNAME}/epoch_{e}_iter_{i}.png'
]
) # note that we attach two images here, the loss plot and
# ...a generated image output from our model
notify.send(msg) # we then send the message
notify_ml.py
In this scenario, every 100 epochs, an email containing all of the above will be sent. Here is one of those emails:
Data Processing and Transfer
This one is slightly less glamorous, but in terms of time consumed, is number one by a long-shot.
We will use the example of bulk data upload to SQL Server using Python (for those of us without BULK INSERT).
At the end of the upload script, we include a simple message notifying us of upload completion.
import os
import notify
from data import Sql # see https://jamescalam.github.io/pysqlplus/lib/data/sql.html
dt = Sql('database123', 'server001') # setup the connection to SQL Server
for i, file in enumerate(os.listdir('../data/new')):
dt.push_raw(f'../data/new/{file}') # push a file to SQL Server
# once the upload is complete, send a notification
# first we create the message
msg = notify.message(
subject='SQL Data Upload',
text=f'Data upload complete, {i} files uploaded.',
)
# send the message
notify.send(msg)
notify_sql.py
If errors are occasionally thrown, we could also add a try-except clause to catch the error, and add it to a list to include in our update and/or completion email.
Financial Modelling
In the case of financial modelling, everything I run is actually pretty quick, so I can only provide you with an ‘example’ use-case here.
We will use the example of a cash-flow modelling tool. In-reality, this process take no more than a 10–20 seconds, but for now let’s assume we’re hot-shot Wall Street quants processing a few million (rather than hundred) loans.
With this email, we may want to include a high-level summary of the analysed portfolio. We can randomly select a few loans and visualize key values over the given time period — giving us a small sample to cross-check model performance is as expected.
end = datetime.datetime.now() # get the ending datetime
# get the total runtime in hours:minutes:seconds
hours, rem = divmod((end - start).seconds, 3600)
mins, secs = divmod(rem, 60)
runtime = '{:02d}:{:02d}:{:02d}'.format(hours, mins, secs)
# now built our message
notify.msg(
subject="Cashflow Model Completion",
text=(f'{len(model.output)} loans processed.\n'
f'Total runtime: {runtime}'),
img=[
'../vis/loan01_amortisation.png',
'../vis/loan07_amortisation.png',
'../vis/loan01_profit_and_loss.png',
'../vis/loan07_profit_and_loss.png'
]
)
notify.send(msg) # and send it
notify_model.py
The Code
All of the functionality above filters from a single script called notify.py.
We will use Outlook in our example code. However, translating this to other providers is incredibly easy, which we will also cover quickly at the end.
There are two Python libraries we need here, email and smtplib.
[**email**](https://docs.python.org/3/library/email.html)— For managing email messages. With this we will setup the email message itself, including subject, body, and attachments.[**smtplib**](https://docs.python.org/3/library/smtplib.html)— Handles the SMTP connection. The simple mail transfer protocol (SMTP) is the protocol used by the majority of email systems, allowing mail to be sent over the internet.’
MIME
The message itself is built using a MIMEMultipart object from the email module. We also use three MIME sub-classes, which we attach to the MIMEMultipart object:
MIMEText— This will contain the email ‘payload’, meaning the text within the email body.MIMEImage— Reasonably easy to guess, this is used to contain images within our email.MIMEApplication— Used for MIME message application objects. For us, these are file attachments.
In addition to these sub-classes, there are also other parameters, such as the Subject value in MimeMultipart. All of these together gives us the following structure.
Let’s take a look at putting these all together.
import os
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from email.mime.application import MIMEApplication
from email.mime.multipart import MIMEMultipart
def message(subject="Python Notification", text="", img=None, attachment=None):
# build message contents
msg = MIMEMultipart()
msg['Subject'] = subject # add in the subject
msg.attach(MIMEText(text)) # add text contents
# check if we have anything given in the img parameter
if img is not None:
# if we do, we want to iterate through the images, so let's check that
# what we have is actually a list
if type(img) is not list:
img = [img] # if it isn't a list, make it one
# now iterate through our list
for one_img in img:
img_data = open(one_img, 'rb').read() # read the image binary data
# attach the image data to MIMEMultipart using MIMEImage, we add
# the given filename use os.basename
msg.attach(MIMEImage(img_data, name=os.path.basename(one_img)))
# we do the same for attachments as we did for images
if attachment is not None:
if type(attachment) is not list:
attachment = [attachment] # if it isn't a list, make it one
for one_attachment in attachment:
with open(one_attachment, 'rb') as f:
# read in the attachment using MIMEApplication
file = MIMEApplication(
f.read(),
name=os.path.basename(one_attachment)
)
# here we edit the attached file metadata
file['Content-Disposition'] = f'attachment; filename="{os.path.basename(one_attachment)}"'
msg.attach(file) # finally, add the attachment to our message object
return msg
notify_message.py
This script, for the most part, is reasonably straightforward. At the top, we have our imports — which are the MIME parts we covered before, and Python’s os library.
Following this define a function called message. This allows us to call the function with different parameters and build an email message object with ease. For example, we can write an email with multiple images and attachments like so:
email_msg = message(
text="Model processing complete, please see attached data.",
img=['accuracy.png', 'loss.png'],
attachments=['data_in.csv', 'data_out.csv']
)
First we initialize the MIMEMultipart object, assigning it to msg.
We then set the email subject using the 'Subject' key.
The attach method allows us to add different MIME sub-classes to our MIMEMultipart object. With this we can add the email body, using the MIMEText sub-class.
For both images img and attachments attachment, we can pass either nothing, a single file-path, or a list of file-paths.
This is handled by first checking if the parameters are None, if they are, we pass. Otherwise, we check the data-type given, is it is not a list, we make it one — this allows us to use the following for loop to iterate through our items.
At this point we use the MIMEImage and MIMEApplication sub-classes to attach our images and files respectively. For both we use os.basename to retrieve the filename from the given file-path, which we include as the attachment name.
SMTP
Now that we have built our email message object, we need to send it.
This is where the smtplib module comes in. The code is again, pretty straight-forward, with one exception.
As we are dealing directly with different email provider’s, and their respective servers, we need different SMTP addresses for each. Fortunately, this is really easy to find.
Type “outlook smtp” into Google. Without even clicking on a page, we are given the server address smtp-mail.outlook.com, and port number 587.
We use both of these when initalizing the SMTP object with smtplib.SMTP — near the beginning of the send function.
import smtplib
import socket
def send(server='smtp-mail.outlook.com', port='587', msg):
# contain following in try-except in case of momentary network errors
try:
# initialise connection to email server, the default is Outlook
smtp = smtplib.SMTP(server, port)
# this is the 'Extended Hello' command, essentially greeting our SMTP or ESMTP server
smtp.ehlo()
# this is the 'Start Transport Layer Security' command, tells the server we will
# be communicating with TLS encryption
smtp.starttls()
# read email and password from file
with open('../data/email.txt', 'r') as fp:
email = fp.read()
with open('../data/password.txt', 'r') as fp:
pwd = fp.read()
# login to outlook server
smtp.login(email, pwd)
# send notification to self
smtp.sendmail(email, email, msg.as_string())
# disconnect from the server
smtp.quit()
except socket.gaierror:
print("Network connection error, email not sent.")
notify_smtp.py
smtp.ehlo() and smtp.starttls() are both SMTP commands. ehlo (Extended Hello) essentially greets the server. starttls informs the server we will be communicating using an encrypted transport level security (TLS) connection. You can learn more about SMTP commands here.
After this we simply read in our email and password from file, storing both in email and pwd respectively.
We then login to the SMTP server with smtp.login, and send the email with smtp.sendmail.
I always send the notifications to myself, but in the case of automated reporting (or for any other reason), you may want to send the email elsewhere. To do this, change the destination_address: smtp.sendmail(email, **_destination_address_**, msg.as_string).
Finally, we terminate the session and close the connection with smtp.quit.
All of this is placed within a try-except statement. In the case of momentary network connection loss, we will be unable to connect to the server. Resulting in a socket.gaierror.
Implementing this try-except statement prevents the program from breaking in the case of a lapse in network connection. How you deal with this may differ, depending on how important it is for the email to be sent.
For me, I use this for ML model training updates and data transfer completion. If an email does not get sent, it doesn’t really matter. So this simple, passive handling of connection loss is suitable.
Putting it Together
Now we have written both parts of our code, we can send emails with just:
# build a message object
msg = message(text="See attached!", img='important.png',
attachment='data.csv')
send(msg) # send the email (defaults to Outlook)
That is all for email notification and/or automation using Python. Thanks to the email and smptlib libraries this is an incredibly easy process to setup.
Common email provider server address and TLS ports. If you have any to add, let me know!
For any processing or training tasks that consume a lot of time, progress updates and notification on completion, is often truly liberating.
I hope this article has been useful to a few of you out there, as always, let me know if you have any questions or suggestions!
Thanks for reading!
Suggest:
☞ Python Tutorials for Beginners - Learn Python Online
☞ Learn Python in 12 Hours | Python Tutorial For Beginners
☞ Complete Python Tutorial for Beginners (2019)
☞ Python Programming Tutorial | Full Python Course for Beginners 2019