Archive for the ‘Programming’ category


Getting Started with VTK for Python

The visualization toolkit (VTK) is a open source library displaying scientific data. VTK is maintained by Kitware, the same company which gave us CMake. VTK is written in C/C++ but it comes with Python bindings and can be installed from https://pypi.org/project/vtk/. In this post, I am going to show how to start using VTK from Python using PyQt5.

Qt has two package for using with Python: PySide2 and PyQt5. PySide2 is the official module for Python but for a long time there was no official module and only PyQt5 was available. You can refer to https://www.learnpyqt.com/blog/pyqt5-vs-pyside2/ to understand the differences (they are mostly same) between two modules. I am going to use PyQt5 but the VTK module itself supports both Qt modules.

VTK provides a QVTKRenderWindowInteractor class which inherits from QWidget, QGLWidget, or any other custom class inherited from QWidget. We will add QVTKRenderWindowInteractor to a QMainWindow and use vtkRenderer to render a Hello, World sphere. To decouple user interface (Qt) and rendering (VTK) I will create a VtkWindow class and use it from a MainWindow which is purely for VTK.

Lets first create the MainWindow:

from PyQt5 import QtCore, QtWidgets
import sys
import VtkWindow

class MainWindow(QtWidgets.QMainWindow):
	def __init__(self, parent=None):
		super(MainWindow, self).__init__(parent)
		self.setWindowState(QtCore.Qt.WindowMaximized)
		
		self.mVtkWindow = VtkWindow.VtkWindow()
		self.setCentralWidget(self.mVtkWindow)
		
if __name__ == '__main__':
	app = QtWidgets.QApplication(sys.argv)
	window = MainWindow()
	window.show()
	app.exec_()

If we comment lines 10 and 11 and run the MainWindow.py, it will display a blank Qt Window. Now lets see how to add VTK support to it by adding a VtkWindow class:

from PyQt5 import QtWidgets
import vtk
import vtkmodules.qt
vtkmodules.qt.QVTKRWIBase = "QGLWidget"
from vtk.qt.QVTKRenderWindowInteractor import QVTKRenderWindowInteractor

# VtkWindow must be derived from QFrame: https://vtk.org/Wiki/VTK/Examples/Python/Widgets/EmbedPyQt
class VtkWindow(QtWidgets.QFrame):
	def __init__(self, parent=None):
		super(QtWidgets.QWidget, self).__init__(parent)
		
		# Create a VTK widget and add it to the QFrame.
		self.setLayout(QtWidgets.QVBoxLayout())
		self.mVtkWidget = QVTKRenderWindowInteractor(self)
		self.layout().addWidget(self.mVtkWidget)
		self.layout().setContentsMargins(0, 0, 0, 0)
		
		# Get the render window and set an interactor.
		self.mRenderWindow = self.mVtkWidget.GetRenderWindow()
		self.mInteractor   = self.mRenderWindow.GetInteractor()
		self.mInteractor.SetInteractorStyle(vtk.vtkInteractorStyleTrackballCamera())
		self.mInteractor.Initialize()
		
		# Create a new renderer and set the background color.
		self.mRenderer = vtk.vtkRenderer()
		self.setBackgroundColor([0.5, 0.5, 0.5])
		self.mRenderWindow.AddRenderer(self.mRenderer)
		
		# Set the Vtk Window title.
		self.mTitleActor = None
		self.setTitle("pyVtkLib Demo")
		
	# Called when QFrame is resized.
	def resizeEvent(self, newSize):
		textSize = [0, 0]
		self.mTitleActor.GetSize(self.mRenderer, textSize)
		
		width  = int( (self.width() - textSize[0]) / 2.0)
		height = self.height() - textSize[1]
		self.mTitleActor.SetPosition(width, height - 10)
		
	def setBackgroundColor(self, color):
		self.mRenderer.SetBackground(color)
		
	def setTitle(self, title):
		if not self.mTitleActor:
			self.mTitleActor = vtk.vtkTextActor()
			self.mTitleActor.GetTextProperty().SetFontFamilyAsString("Georgia")
			self.mTitleActor.GetTextProperty().SetFontSize(30)
			self.mTitleActor.GetTextProperty().SetColor([1, 0, 0])
			self.mTitleActor.SetInput(title)
			self.mTitleActor.SetPosition(0, 0)
			self.mRenderer.AddActor(self.mTitleActor)
		else:
			self.mTitleActor.SetInput(title)
	

VTK module for Python comes with a QVTKRenderWindowInteractor class which by default inherits from QWidget for PyQt5. In lines 4-5, we first change it to to use QGLWidget so that rendering will be done using OpenGL instead of software renderer. Next, we create a class called VtkWindow which inherits from QWidget so that it can be use from Qt UI. Note, that it is recommended to inherit from QFrame and not QWidget as QVTKRenderWindowInteractor cannot be reparented. More discussion on this topic can be found at EmbedPyQt example on VTK website. Next, we create an instance of QVTKRenderWindowInteractor and add it to VtkWindow class through a QVBoxLayout.

After that it is usual VTK stuff of creating a vtkRenderingWindow, vtkRenderWindowInteractor, and vtkRenderer. I prefer to use vtkInteractorStyleTrackballCamera which I find far more intuitive than the default vtkInteractorStyleJoystickCamera.

I render scene title at the top-middle of the screen and in order to place it here I listen to QFrame::resizeEvent to determine te current width and height of the QFrame.

Run the MainWindow.py from a terminal and it will display a windows with text pyVtkLib Demo printed in the middle-center of the window. In the next tutorial I will show how to measure and show frames per second to the VtkWindow.

The code from this tutorial and any other future enhancements I will do will be available from saurabhg17/pyVtkLib repository at GitHub.


QWidgetLogger: A Qt logging device for SLogLib

SLogLib is a flexible logging library written in C++. It comes with three logging devices: console, file, and memory but often there is a need to show log directly on the user interface. I heavily use Qt for all projects requiring user interface, so I wrote a logging device for QWidget. It is available in SLogLib repository in the Contrib/QWidgetLogger folder.

Here is how to use the QWidgetLogger:

QWidgetLogger* logger = new QWidgetLogger(new HtmlFormatter(), "WidgetLogger");
SLogLib::addLoggingDevice(logger);

Once above code is added, any of the SLOGLIB_LOG_MSG_* logging macros will write to the QWidgetLogger as well as any other loggers added to SLogLib. QWidgetLogger internally uses SLogTextEdit class derived from QTextEdit. Instance of SLogTextEdit used by QWidgetLogger can be retrieved by QWidgetLogger::widget() function. This instance should be added to the UI to show the logging messages. QDockWidget is a good choice to show logging widget with QMainWindow.

SLogTextEdit sets a monospace 10 point font and it can be changed using style sheet. The color and style of the messages logged can also be changed using the Formatter. The HtmlFormatter used in the example above define different colors for different types of logging messages using HTML codes.

In Qt, UI elements can only be updated from the main thread but the logging messages might come from any thread. So SLogTextEdit checks if the message was posted from the main thread or some other worker thread. If the message was posted from a worker thread, SLogTextEdit emits a signal to only of its own private slot and updates itself in the slot. In Qt slots always run in the context of the main thread. This method works well but signal and slot mechanism is slow and update to widget lags while logging too many messages in a short period of time.


Added Multi-threading Support in SLogLib

Recently, I worked on a project which made heavy use of C++ threads. To use SLogLib in this project I added multi-threaded support in SLogLib with the help of c++ 11 std::mutex and std::lock_guard. Over the last few months multi-threaded support in SLogLib has been extensively tested and there are no known bugs.

All threading support is located in LoggingManager.cpp. The functions which modify internal state in LoggingManager are protected by std::mutex. There is a support for building call stack through the use of SLOGLIB_ADD_TO_CALLSTACK macro. In the latest build, there is a separate callstack for each thread.

Checkout the latest commit from https://github.com/saurabhg17/SLogLib.


Error loading numpy arrays

If you received “Object arrays cannot be loaded when allow_pickle=False” while loading a numpy array using numpy.load() it is because numpy has changed the default loading behaviour since version 1.16.3. If you are using numpy version newer than this, at many places on internet it is advised to simply downgrade the numpy version. This is not the correct solution at all. From the numpy documentation:

allow_pickle : bool, optional
Allow loading pickled object arrays stored in npy files. Reasons for disallowing pickles include security, as loading pickled data can execute arbitrary code. If pickles are disallowed, loading object arrays will fail. Default: False
Changed in version 1.16.3: Made default False in response to CVE-2019-6446.

Thus, the correct solution is to pass allow_pickle=True to the numpy.load function. However, this should be used carefully and ideally only with the files you have previously saved yourself since picking in python can execute arbitrary code thereby compromising the system security.


String Selection Widget for Qt5

Some time back, I developed a data entry application in Qt5. One of the requirements was to let the user select a single string from a predefined list of string. I developed a custom widget called SStringSelector for this purpose. SStringSelector has two views: display and selection. The display view presents the currently selected string (blank if no string is selected), and a push button. To select a string, the user clicks on the button which presents the user with the selection dialog. The selection dialog consists of a list of string in an QListWidget and the user can select one of them by double-clicking a string. If the list of strings are long, the user can filter them using a filter QLineEdit present above the QListWidget.

SStringSelector is distributed as a part of QtUtils repository hosted on Github. The SStringSelector widget is really simple to use. Simple add the SStringSelector.h and SStringSelector.cpp files in your project and add an instance of SStringSelector in the layout of your app.

Below are some screenshots of the widget under Windows:

The SStringSelector Widget.
Selection Dialog of the SStringSelector Widget.
Filtering Strings in the Selection Dialog.

Color Picker Widget for Qt5

Qt5 support standard dialogs such as QFileDialog, QFontDialog, and QColorDialog, however, it does not provide a color picker to allow a user to pick a color. Recently, I need a color picker for one of my projects and I implemented a simple color picker widget.

SColorPicker is available from Github as a part of QtUtils repository. To use SColorPicker, add the header and cpp files directly in your project. Then, simply add an instance of SColorPicker in a layout. SColorPicker will appear as 16×16 pixels colored square in the layout. If you need a different size, change it in the SColorPicker's constructor. When a user double-clicks on the colored square, the system’s color dialog will appear allowing the user to choose a color. The selected color can be obtained from color() function or by connecting to colorPicked() signal.

Below are the screenshots of the SColorPicker_Demo and system color dialog present to the user on Windows 10 computer.


Fill Disk Partition

Recently, I had to give away a computer with couple of disks in it. I wanted to securely erase data on these disks as I stored personal sensitive information on them. Using a program such as DBAN was not an option as I was not allowed to remove the operating system from the computer. My goal was to simply overwrite free space from all the partitions. I couldn’t find anything I liked so I ended up writing a simple tool called FillPartition in python.

FillPartition is hosted on Github at https://github.com/saurabhg17/FillPartition. It is really easy to use with just one mandatory argument (the path of the partition) and one optional argument (–outputDir, -od) the directory in the partition where files should be written. FillPartition writes 1GB files filled with 0 bytes until the free space is less than 1GB and then write one final file of the size equal to the remaining free space.

Below is a screenshot of a run of FillPartition on Windows


Search Box using QLineEdit

This week, at work I had to implement a search box for a software I am working on. The search box is to filter some data dynamically as user types a query. I wanted to show a clear (cross) icon at the right side of the search box so that user can clear the results instead of selecting the current query and deleting it manually. Lastly, for clarity I wanted to show a search icon on the left side of search box. The search box looks like this:

Screenshot of the Search box implemented using QLineEdit

After the user enters a query a clear icon appears on the right. The clear icon is in fact a button and clicking it will clear the current search.
Screenshot of the Search box with keywords implemented using QLineEdit

It is really easy to make this search box using QLineEdit. We need only the following three lines of code:

QLineEdit* _lineEdit = new QLineEdit();
_lineEdit->setClearButtonEnabled(true);
_lineEdit->addAction(":/resources/search.ico", QLineEdit::LeadingPosition);
_lineEdit->setPlaceHolderText("Search...");

// add _lineEdit to your widget

Line 2 enables the clear button which adds the clear action and cross icon to the right. Line 3 adds another action with a search icon to the left of the QLineEdit. We don’t listen to this action as it is merely decorative. Line 4 adds a placeholder text which is shown in the QLineEdit but is cleared as soon as user starts typing.

We only connect textChanged(const QString&)  signal which is emitted both when a user clicks on the cross icon and when he enters a search query.

 




Markdown to PDF Converter

Few weeks ago, I published SLogLib (a cross-platform logging library) on GitHub. I wrote the user manual in a readme.md file as is the standard practice at GitHub. However, since most of users don’t have markdown viewers installed on their machines they would either need to access GitHub repository or would have to convert it to more popular format such as PDF or perhaps HTML. For many users going online is becoming standard practice to access documentation but I prefer offline manuals. Thus, I wanted to ship a PDF manual along with the code.

I searched high and low for a standalone tool to convert markdown to PDF but surprisingly there are not a lot of options out there. The first tool I came across was GitPrint. It is conceptually innovative and straightforward to use with GitHub. Just add /your_user_name/repository_name at the end of http://gitprint.com and it prints the readme.md in the repository to PDF. The PDF generated is of good quality but there are no styling options. Also, it failed to include images in the PDF so I had to kept looking. One of the frequently recommended tool is PanDoc, which is a swiss-army knife to convert files from one markup format into another. However, in my experience it doesn’t do a good job of converting markdown to PDF. Another popular tool online is a markdown-pdf package for Node.js. Since, I have no prior experience with Node.js I haven’t tried it yet.

Earlier this year, I bought a MacBook Pro and installed a markdown editor called MacDown. It is really nice tool with side-by-side rendering of markup and HTML. It can export markdown as PDF and produces very good quality PDF’s. It also supports lots of styling options as well as a CSS to customize PDF generation. In the end, I used it to generate PDF for SLogLib.

Even though I had a PDF for SLogLib, I wanted to find/build a cross-platform tool to convert markdown to PDF.

The basic idea to convert markdown to PDF is simple. First convert markdown to HTML and then print HTML to PDF. I used hoedown to convert markdown to PDF because of several reasons:

  1. First and foremost it is cross-platform and compiles as a standalone binary for all three main platforms: Windows, Linux, and OSX.
  2. MacDown uses it too and I was quite happy with its rendering.
  3. It supports not only standard markdown but also several non-standard extensions.

To converted HTML to PDF one of the most popular tool I came across was wkhtmltopdf. It is also cross-platform and complies into standalone binaries for all popular platforms. In fact, it is possible to download the pre-built library right from its website. Wkhtmltopdf uses a modified version of webkit shipped with Qt. It uses webkit to render the html and print to PDF. However, while testing I found that on a Windows 7 machines there is a serious problem with font kerning. It has been reported by a lot of users but I haven’t found a solution to fix it. Wkhtmltopdf would have been ideal as I could simply write a command line and/or GUI tool wrapping the functionality of hoedown and Wkhtmltopdf.

Screenshot of markdown to PDF generated from MacDown

Screenshot of PDF generated from MacDown.

Screenshot of markdown to PDF generated wkhtmltopdf

Screenshot of PDF generated from wkhtmltopdf.

I could not find any other standalone cross-platform tool to convert HTML to PDF. So, for now I decided to use dompdf which is written in PHP. Once I started used PHP I thought why not make it a web based tool. This would allow me to learn about SEO which I have been promising myself to learn one day :). The tools is hosted at http://markdown2pdf.com. At the moment it doesn’t appear in first five pages in Google search for “markdown to pdf” or “markdown 2 pdf”. I am playing with various SEO tools and techniques and hope to get it within first five pages.

My quest for a standalone tool is not yet complete. I will try to find a solution for wkhtmltopdf kerning issue or find another standalone cross-platform tool for converting from HTML to PDF. I will update with my findings on this blog.




Cross-platform high-resolution timer

Often there is a need to estimate the time it takes for a piece of code to run. This is useful not only for debugging but also for reporting the execution time of lengthy tasks to the user.

On Windows, QueryPerformanceFrequency() and QueryPerformanceCounter() can be used to determine the execution time of a code. QueryPerformanceFrequency() returns the frequency of the current performance counter in counts per second and QueryPerformanceCounter() returns a high resolution (<1µs) time stamp. Together they can be used to determine time it takes to run a piece of code is:

LARGE_INTEGER _frequency;
QueryPerformanceFrequency(&_frequency);

LARGE_INTEGER _start;
QueryPerformanceCounter(&_start);

// Code which takes a long time to run.

LARGE_INTEGER _stop;
QueryPerformanceCounter(&_stop);

double _intervalInSeconds = (_stop.QuadPart - _start.QuadPart) / _frequency.QuadPart;

On Linux, clock_gettime can be used to get a time interval with a resolution of nano-seconds. clock_gettime() requires two arguments: clockid_t and timespec structure. To build a timer, CLOCK_MONOTONIC is a good choice for clockid_t as the time is guaranteed to be monotonically increasing. timespec structure have two field: tv_sec (time in seconds) and tv_nsec (time in nanoseconds). Code to determine the time it takes to run a piece of code is:

struct timespec _start;
clock_gettime(CLOCK_MONOTONIC, &_start);

// Code which takes long time to run.

struct timespec _stop;
clock_gettime(CLOCK_MONOTONIC, &_stop);

double _intervalInseconds = (_stop.tv_sec + _stop.tv_nsec*1e-9) - (_start.tv_sec + _start.tv_nsec*1e-9);

I have written a simple class which can be user on both windows and Linux. It has the following interface:

class Timer
{
public:

    enum TimeUnit
    {
        TimeInSeconds = 1,
        TimeInMilliSeconds = 1000,
        TimeInMicroSeconds = 1000000
    };

public:

    Timer();
    ~Timer();

    // On Windows, returns true if high performance timer is available.
    // On Linux, always returns true.
    bool IsTimerAvailable();

    // Start the timer.
    void Start();

    // Stop the timer and return the time elapsed since the timer was started.
    double Stop(TimeUnit timeUnit = TimeInMilliSeconds);

    // Get the time elapsed since Start() was called.
    double TimeElapsedSinceStart(TimeUnit timeUnit = TimeInMilliSeconds);

    // Get the total time elapsed between Start() and Stop().
    double TotalTimeElasped(TimeUnit timeUnit = TimeInMilliSeconds);
};

You can download the code from the following links:
Timer.h
Timer.cpp
Timer_Unix.cpp
Timer.zip