This post explains how to generate management-quality PDF or HTML reporting directly from Jupyter Notebook. With this technique, I reduced to zero the most irritating part of my projects: copy-pasting diagrams into PowerPoint.
The workbook pasted below demonstrates how this can be done.
This workbook makes an integral part of this article: OnData.blog: I stopped writing reports
Mystery Plot¶
Have a look below. The patterns are intriguing. I think there is some regularity in the way the dots are organized. Let’s discuss it at the next business meeting.
abracadabra()
And by the way, this is how many dots there are in the diagram:
len(x)
3000
That much for business readers. Now, programmers will be surprised that this page is in fact a regular Jupyter Notebook, rendered as HTML. Wait a second… we used a method abracadabra(), and we used a variable x. But where are they defined? Well… magic. Read on.
This workbook makes an integral part of this article: OnData.blog: I stopped writing reports
I hate writing reports
I hate reporting. I love creative research and discovering truth in data. But then the process of collecting the facts into a document or presentation is dull and boring. The worst part comes when diagrams need to be copied in PowerPoint. If I spot an error in the diagram, I need to generate it again and paste again.
Over the past week I had some fun analyzing the client’s historical data. The final PowerPoint was rich with details: project health analysis, predictive analytics, anomaly detection, workload allocation, automation analysis, trends analysis, and so on. The client appreciated the results and asked whether I could run similar analysis on five other projects. I could, but the perspective of copy-pasting one hundred diagrams into five more PowerPoints was dreadful.
Then I thought, couldn’t I possibly generate the final report directly from JupyterLab Notebook?
Why not simply save the notebook as PDF?
Notebooks can be exported as PDF, webPDF or HTML, using File / Export Notebook As menu option.
So in principle, with some discipline and keeping the notebook tidy, you could organize the notebook in such a way that it serves as the final report.
But my notebooks document my research process and their structure is not suitable to the audience of my end reports: managers, who are not interested in my thought process, neither in the source code, but the business conclusions.
So it would be optimal to maintain two notebooks. One to document the research process, and the other for the reporting. I found a simple and efficient way for this.
Ultimate solution: share kernel between two notebooks
Below is the content of my research notebook A. It contains some code, which I don’t want the business users to see. It defines some methods and variables. Then, after the work is done, I generate all diagrams in the workbook B, which you have already seen above. The only trick is to share the Jupyter kernel between the both.
This workbook makes an integral part of this article: OnData.blog: I stopped writing reports
# Here we generate two Gaussian distributions
import numpy as np, matplotlib.pyplot as plt, seaborn as sns
sns.set()
x = np.random.normal(size = 30000)
y = np.random.normal(size = 30000)
# this diagram is okay for the research phase
sns.scatterplot(x = x,y = y, color = 'black', s = 1, edgecolor =None, alpha = 1)
# For the business report, let's prepare a modified diagram that better fits the audience.
def abracadabra():
sns.scatterplot(x = x[:3000],y = y[:3000], color = 'black', s = 20, edgecolor ='red', alpha = 1)
Explanation:
This workbook A, together with workbook B, demonstrates how to generate management-quality reports directly from Jupyter Notebook.
This workbook contains diagrams together with some code. I don’t want to see this code in the final report. I will achieve this with shared kernel. Here is how:
- This research workbook A serves to define all your variables and methods. Open it in JupyterLab and run it.
- In the same JupyterLab, open the reporting notebook B in a separate tab. Change the kernel and “steal” the environment from the research notebook A (menu Kernel/Change Kernel). Now the notebook B can use the variables and results computed in notebook A.
- Run the reporting notebook B, so the diagrams get rendered
- Export the notebook B to HTML or PDF using menu File/Export workbook as .
There is also a way to export workbook B to PDF without showing any source code cells whatsoever, so audience won’t even know that abracadabra() was being executed. This can be done at the shell level. This command worked for me at the anaconda prompt:
_jupyter nbconvert notebook_name.ipynb –no-input –to webpdf –no-prompt –output outputfilename
Why is it so cool?
It is very cool.
Firstly, It will happen quite often that I spot a mistake in the research process. Then I can modify workbook A, and when ready, the corrected report will be generated from workbook B. No need to manually improve the report.
Secondly, I may need to deliver similar reports for several projects. Then I will only adapt workbook A for every project. Workbook B can be reused without any change, and all the reports will be generated the same way.
Most importantly, with this method I will not need to paste a single diagram into PowerPoint.
This is how I stopped writing reports, and so can you.
Note: the source code for both notebooks is available in GitHub: shared-kernel-A and shared-kernel-B.