Display FPS for VTK on Python

In the last post, I discussed how to get started with VTK on Python. In this post, I will show how to add support to show frames per second (FPS). The idea to calculate FPS is straight forward: keep track of the number of frames (N) that were rendered in last T seconds. Then fps defined a N/T fps.

To calculate FPS we will add an observer to the EndEvent command of the vtkRenderer. In the callback function, we will count the number of frames rendered in the last T seconds and calculate FPS. Here is the complete code of the FpsObserver:

import vtk
from timeit import default_timer as timer

class FpsObserver:
	def __init__(self, renderer, x=0, y=0):
		self.mRenderer = renderer
		self.mRenderer.AddObserver(vtk.vtkCommand.EndEvent, self)
		
		self.ActorPosX = x
		self.ActorPosY = y
		
		self.mFrameCount    = 0         # Number of frames collected since last FPS was calculated.
		self.mStartTime     = timer()   # The last time FPS was calculated.
		self.mFpsUpdateRate = 1         # How often to update FPS in seconds.
		
		self._createFpsTextActor()
	
	def setPosition(self, x, y):
		self.ActorPosX = x
		self.ActorPosY = y
		self.mFpsActor.SetPosition(self.ActorPosX, self.ActorPosY)
	
	def __call__(self, caller, event):
		if event == "EndEvent":
			self.mFrameCount = self.mFrameCount + 1
			
			if timer() - self.mStartTime > self.mFpsUpdateRate:
				_currentTime     = timer()
				_duration        = _currentTime - self.mStartTime
				
				_fps = self.mFrameCount/_duration
				print("fps={:.3f}".format(_fps))
				self.mFpsActor.SetInput("FPS: {:.2f}".format(_fps))
				
				self.mStartTime  = _currentTime
				self.mFrameCount = 0
				
	def _createFpsTextActor(self):
		self.mFpsActor = vtk.vtkTextActor()
		self.mFpsActor.GetTextProperty().SetFontFamilyAsString("Georgia")
		self.mFpsActor.GetTextProperty().SetFontSize(20)
		self.mFpsActor.GetTextProperty().SetColor([1, 1, 1])
		self.mFpsActor.SetPosition(self.ActorPosX, self.ActorPosY)
		self.mRenderer.AddActor(self.mFpsActor)

To use FpsObserver, we just need to initialize it as self.mFpsObserver = FpsObserver.FpsObserver(self.mRenderer). That’s it, this will display the FPS for last one seconds!


Getting Started with VTK for Python

The visualization toolkit (VTK) is a open source library displaying scientific data. VTK is maintained by Kitware, the same company which gave us CMake. VTK is written in C/C++ but it comes with Python bindings and can be installed from https://pypi.org/project/vtk/. In this post, I am going to show how to start using VTK from Python using PyQt5.

Qt has two package for using with Python: PySide2 and PyQt5. PySide2 is the official module for Python but for a long time there was no official module and only PyQt5 was available. You can refer to https://www.learnpyqt.com/blog/pyqt5-vs-pyside2/ to understand the differences (they are mostly same) between two modules. I am going to use PyQt5 but the VTK module itself supports both Qt modules.

VTK provides a QVTKRenderWindowInteractor class which inherits from QWidget, QGLWidget, or any other custom class inherited from QWidget. We will add QVTKRenderWindowInteractor to a QMainWindow and use vtkRenderer to render a Hello, World sphere. To decouple user interface (Qt) and rendering (VTK) I will create a VtkWindow class and use it from a MainWindow which is purely for VTK.

Lets first create the MainWindow:

from PyQt5 import QtCore, QtWidgets
import sys
import VtkWindow

class MainWindow(QtWidgets.QMainWindow):
	def __init__(self, parent=None):
		super(MainWindow, self).__init__(parent)
		self.setWindowState(QtCore.Qt.WindowMaximized)
		
		self.mVtkWindow = VtkWindow.VtkWindow()
		self.setCentralWidget(self.mVtkWindow)
		
if __name__ == '__main__':
	app = QtWidgets.QApplication(sys.argv)
	window = MainWindow()
	window.show()
	app.exec_()

If we comment lines 10 and 11 and run the MainWindow.py, it will display a blank Qt Window. Now lets see how to add VTK support to it by adding a VtkWindow class:

from PyQt5 import QtWidgets
import vtk
import vtkmodules.qt
vtkmodules.qt.QVTKRWIBase = "QGLWidget"
from vtk.qt.QVTKRenderWindowInteractor import QVTKRenderWindowInteractor

# VtkWindow must be derived from QFrame: https://vtk.org/Wiki/VTK/Examples/Python/Widgets/EmbedPyQt
class VtkWindow(QtWidgets.QFrame):
	def __init__(self, parent=None):
		super(QtWidgets.QWidget, self).__init__(parent)
		
		# Create a VTK widget and add it to the QFrame.
		self.setLayout(QtWidgets.QVBoxLayout())
		self.mVtkWidget = QVTKRenderWindowInteractor(self)
		self.layout().addWidget(self.mVtkWidget)
		self.layout().setContentsMargins(0, 0, 0, 0)
		
		# Get the render window and set an interactor.
		self.mRenderWindow = self.mVtkWidget.GetRenderWindow()
		self.mInteractor   = self.mRenderWindow.GetInteractor()
		self.mInteractor.SetInteractorStyle(vtk.vtkInteractorStyleTrackballCamera())
		self.mInteractor.Initialize()
		
		# Create a new renderer and set the background color.
		self.mRenderer = vtk.vtkRenderer()
		self.setBackgroundColor([0.5, 0.5, 0.5])
		self.mRenderWindow.AddRenderer(self.mRenderer)
		
		# Set the Vtk Window title.
		self.mTitleActor = None
		self.setTitle("pyVtkLib Demo")
		
	# Called when QFrame is resized.
	def resizeEvent(self, newSize):
		textSize = [0, 0]
		self.mTitleActor.GetSize(self.mRenderer, textSize)
		
		width  = int( (self.width() - textSize[0]) / 2.0)
		height = self.height() - textSize[1]
		self.mTitleActor.SetPosition(width, height - 10)
		
	def setBackgroundColor(self, color):
		self.mRenderer.SetBackground(color)
		
	def setTitle(self, title):
		if not self.mTitleActor:
			self.mTitleActor = vtk.vtkTextActor()
			self.mTitleActor.GetTextProperty().SetFontFamilyAsString("Georgia")
			self.mTitleActor.GetTextProperty().SetFontSize(30)
			self.mTitleActor.GetTextProperty().SetColor([1, 0, 0])
			self.mTitleActor.SetInput(title)
			self.mTitleActor.SetPosition(0, 0)
			self.mRenderer.AddActor(self.mTitleActor)
		else:
			self.mTitleActor.SetInput(title)
	

VTK module for Python comes with a QVTKRenderWindowInteractor class which by default inherits from QWidget for PyQt5. In lines 4-5, we first change it to to use QGLWidget so that rendering will be done using OpenGL instead of software renderer. Next, we create a class called VtkWindow which inherits from QWidget so that it can be use from Qt UI. Note, that it is recommended to inherit from QFrame and not QWidget as QVTKRenderWindowInteractor cannot be reparented. More discussion on this topic can be found at EmbedPyQt example on VTK website. Next, we create an instance of QVTKRenderWindowInteractor and add it to VtkWindow class through a QVBoxLayout.

After that it is usual VTK stuff of creating a vtkRenderingWindow, vtkRenderWindowInteractor, and vtkRenderer. I prefer to use vtkInteractorStyleTrackballCamera which I find far more intuitive than the default vtkInteractorStyleJoystickCamera.

I render scene title at the top-middle of the screen and in order to place it here I listen to QFrame::resizeEvent to determine te current width and height of the QFrame.

Run the MainWindow.py from a terminal and it will display a windows with text pyVtkLib Demo printed in the middle-center of the window. In the next tutorial I will show how to measure and show frames per second to the VtkWindow.

The code from this tutorial and any other future enhancements I will do will be available from saurabhg17/pyVtkLib repository at GitHub.


Goodbye Launchy!

I have been using Launchy for last 15 years since it was first released in 2005. I has been an awesome companion since then and saved me tons of time. Simple press Alt + Space and enter the name of the program you want execute and voila! It allowed skins and specifying directories to include in search and filter the folders based on extensions. Launchy had everything I could have asked. However, it doesn’t work with Windows 10 store apps and it was last updated in 2010 so there are no new updated or patches since then.

Recently, Microsoft released Power Toys as opensource tool on Github. One of the included tools is PowerToys Run. It has the same core functions plus more other added goodies such as “Open containing folder”, “Run as administrator”, and “Open path in console” which I fund quite useful. My only complaint with PowerToys Run is that it doesn’t allow customs paths and instead uses Windows Search Index.


QWidgetLogger: A Qt logging device for SLogLib

SLogLib is a flexible logging library written in C++. It comes with three logging devices: console, file, and memory but often there is a need to show log directly on the user interface. I heavily use Qt for all projects requiring user interface, so I wrote a logging device for QWidget. It is available in SLogLib repository in the Contrib/QWidgetLogger folder.

Here is how to use the QWidgetLogger:

QWidgetLogger* logger = new QWidgetLogger(new HtmlFormatter(), "WidgetLogger");
SLogLib::addLoggingDevice(logger);

Once above code is added, any of the SLOGLIB_LOG_MSG_* logging macros will write to the QWidgetLogger as well as any other loggers added to SLogLib. QWidgetLogger internally uses SLogTextEdit class derived from QTextEdit. Instance of SLogTextEdit used by QWidgetLogger can be retrieved by QWidgetLogger::widget() function. This instance should be added to the UI to show the logging messages. QDockWidget is a good choice to show logging widget with QMainWindow.

SLogTextEdit sets a monospace 10 point font and it can be changed using style sheet. The color and style of the messages logged can also be changed using the Formatter. The HtmlFormatter used in the example above define different colors for different types of logging messages using HTML codes.

In Qt, UI elements can only be updated from the main thread but the logging messages might come from any thread. So SLogTextEdit checks if the message was posted from the main thread or some other worker thread. If the message was posted from a worker thread, SLogTextEdit emits a signal to only of its own private slot and updates itself in the slot. In Qt slots always run in the context of the main thread. This method works well but signal and slot mechanism is slow and update to widget lags while logging too many messages in a short period of time.


Added Multi-threading Support in SLogLib

Recently, I worked on a project which made heavy use of C++ threads. To use SLogLib in this project I added multi-threaded support in SLogLib with the help of c++ 11 std::mutex and std::lock_guard. Over the last few months multi-threaded support in SLogLib has been extensively tested and there are no known bugs.

All threading support is located in LoggingManager.cpp. The functions which modify internal state in LoggingManager are protected by std::mutex. There is a support for building call stack through the use of SLOGLIB_ADD_TO_CALLSTACK macro. In the latest build, there is a separate callstack for each thread.

Checkout the latest commit from https://github.com/saurabhg17/SLogLib.


Error loading numpy arrays

If you received “Object arrays cannot be loaded when allow_pickle=False” while loading a numpy array using numpy.load() it is because numpy has changed the default loading behaviour since version 1.16.3. If you are using numpy version newer than this, at many places on internet it is advised to simply downgrade the numpy version. This is not the correct solution at all. From the numpy documentation:

allow_pickle : bool, optional
Allow loading pickled object arrays stored in npy files. Reasons for disallowing pickles include security, as loading pickled data can execute arbitrary code. If pickles are disallowed, loading object arrays will fail. Default: False
Changed in version 1.16.3: Made default False in response to CVE-2019-6446.

Thus, the correct solution is to pass allow_pickle=True to the numpy.load function. However, this should be used carefully and ideally only with the files you have previously saved yourself since picking in python can execute arbitrary code thereby compromising the system security.


Solution to Kaggle’s Dogs vs. Cats Challenge using Logistic Regression

In the previous post, I discussed a solution to Kaggle’s Dogs vs. Cats Challenge using Convolutional Neural Networks. CNN’s takes time to train and I tried a number of different network models and various values for hyperparameters before achieving 94% accuracy. This was very time consuming and it took around two days to determine the best network model and values of the hyperparameters. I used grid-search with the help of TrainCNN.py [1] to tune the value of hyperparameters. One run of TrainCNN.py for grid-search took few hours and since I was unable to do anything related to CNN, I decided to try Logistic Regression on another machine to solve the problem. I used LogisticRegressionCV from Scikit-learn which is the cross-validated version of the LogisticRegression function. I am not going to discuss the code in this blog post as it is straightforward implementation and instead encourage you to read it from LogisticRegression.py in my Exploring Deep Learning repository at Github.

Kaggle’s dogs vs. cats dataset has 25,000 images in two equal classes of dogs and cats. I used 15,000 (7,500 each for dogs and cats) randomly selected images for fitting model and 5,000 images (2,500 each for dogs and cats) for validation.

There are two parameters for processing the dataset itself: image size and whether to standardizing images or not. For logistic regression there is choice of solver and a hyperparameter called Cs which describes the strength of the regularization. Smaller values of Cs specifies stronger regularization. I did grid-search for optimal solution for these parameters and below are the results:

Solver  ImageSize  Rescale  TrainingAcc  ValidationAcc  TimeToFit (s)  Memory (GB)
lbfgs75True67.661.8308.713.5
lbfgs100True70.161.3544.823.6
lbfgs125True72.360.6857.936.5
 
sag75True67.661.91222.613.2
sag100True70.161.32255.523.2
sag125True72.660.53572.636.1
 
lbfgs125False81.757.3944.436.5
sag125False84.858.44072.136.1
 
lbfgs125True68.162.03635.836.5

Solver

Sklearn recommends using liblinear for a smaller dataset and sag or saga for larger dataset. However, the default solver is lbfgs for logistic regression. Since dogs vs. cats dataset is relatively large for logistic regression, I decided to compare lbfgs and sag solvers. Comparing rows 1-3 with 4-6, we can see that although the training and validation accuracy is same for both lbfgs and sag solvers, the sag solver is about four times slower than lbfgs solver. Thus, sklean has a good default value of lbfgs as solver for logistic regression.

Image Size

If we compare image size for any one solver (rows 1-3 or 4-6) we can see that as the image size increases, training accuracy increases from 67.6% to 72.6%. However, the validation accuracy stays roughly the same at 61-62%. This indicates that model is being over-fitting over training samples. In the regularization section, we will see how to handle overfitting by adjusting the regularization strength.

Image Normalization

Sklearn recommends that features should be approximately of the same scale. “Note that [for] ‘sag’ and ‘saga’ fast convergence is only guaranteed on features with approximately the same scale. You can preprocess the data with a scaler from sklearn.preprocessing” [2]. I used sklean.preprocessing.StandardScaler to normalized both training and validation data. StandardScaler transform the data so that each feature has a zero mean and unit standard deviation. Looking at the rows 7 and 8, we can see that without image normalization both lbfgs and sag massively overfits the training data with the training accuracy of 82% and 85%, respectively and the validation accuracy of only 57% and 58%. Both solvers are also about three times slower then when images were normalized. This clearly highlights the importance of the feature normalization.

Regularization

Once I decided on the solver (lbfgs), image size (125), and that images should be normalized, I fine tuned for regularization strength (Cs). I used L2 regularization since lbfgs supports only L2 regularization. To use L1 regularization we have to use saga solver but since sag and saga are so much slower than lbfgs I decided not to try it out. LogisticRegressionCV in sklearn supports grid-search for hyperparameters internally, which means we don’t have to use model_selection.GridSearchCV or model_selection.RandomizedSearchCV. LogisticRegressionCV has a parameter called Cs which is a list all values among which the solver will find the best model. I used Cs = [1e-12, 1e-11, …, 1e11, 1e12]. The results for fine tuning is presented in the last row (row 9) in the table above. It can be seen that the training accuracy has dropped from 72.3% to 68.1% while validation accuracy has increased from 60.6% to 62%. This, tuning for regularization strength does indeed decrease the degree of overfitting the training data.

Conclusions

In this article, I presented results for image classification for Kaggle’s dogs vs. cats dataset using logistic regression. The classifier achieved an accuracy of 62% on validation images. It may be possible to achieve higher accuracy by further tuning image size, preprocessing images, using a grayscale image instead of RGB color images, using a different value of regularization strength, or using both L1 and L2 regularization. I choose not to further explore since the memory requirements for logistic regression in sklearn is very large (last column in the table above).

References

  1. https://github.com/saurabhg17/ExploringDeepLearning
  2. https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html

Solution to Kaggle’s Dogs vs. Cats Challenge using Convolutional Neural Networks

Dogs vs. cats challenge [1] from Kaggle ended in Jan 2014 but it is still extremely popular for getting started in deep learning. This is because of two main reasons: the data set is small (25,000 images taking up about 600MB), and it is relatively easy to get a good score.

There are many many online articles discussing on how pre-process data , design a CNN model and finally train the model. So, in this post I am not going to discuss the implementation details. Instead, I am simply going to report my results using a custom designed model and transfer learning. I used Tensorflow and tf.keras with Python and it is available from my Exploring Deep Learning repository [2] at Github.

Learning using a Custom Model

Note that this is my best attempt and not the first attempt. I used four blocks of 2D convolution layers followed by max pooling. In the end, I used two dense layers and a softmax layer as output. I also used dropout layers and image augmentation. The exact command line for training this model is:

TrainCNN.py --cnnArch Custom --classMode Categorical --optimizer Adam --learningRate 0.0001 --imageSize 224 --numEpochs 30 --batchSize 16 --dropout --augmentation --augMultiplier 3

The CNN model is given below:

--------------------------------------------------------------- 
Model: "Custom"
---------------------------------------------------------------   
Layer (type)                 Output Shape              Param #

conv2d (Conv2D)              (None, 224, 224, 32)      896
max_pooling2d (MaxPooling2D) (None, 112, 112, 32)      0

conv2d_1 (Conv2D)            (None, 112, 112, 64)      18496
max_pooling2d_1 (MaxPooling2 (None, 56, 56, 64)        0
 
conv2d_2 (Conv2D)            (None, 56, 56, 128)       73856
max_pooling2d_2 (MaxPooling2 (None, 28, 28, 128)       0
 
conv2d_3 (Conv2D)            (None, 28, 28, 256)       295168
max_pooling2d_3 (MaxPooling2 (None, 14, 14, 256)       0
 
flatten (Flatten)            (None, 50176)             0
dense (Dense)                (None, 512)               25690624
dense_1 (Dense)              (None, 256)               131328
dense_2 (Dense)              (None, 2)                 514
=============================================================== 
Total params: 26,210,882
Trainable params: 26,210,882
Non-trainable params: 0
--------------------------------------------------------------- 

The above model was trained on 15,000 (7,500 each for dogs and cats) randomly chosen images from the Kaggle data set and validated with a separate 5,000 (2,500 each for dogs and cats) images. The model achieved 94% accuracy after 24 epochs. It took about 4 hours of training on my PC with NVidia GeForce GTX 1050 with 2GB of RAM.

Cross-entropy loss for training and validation and the classification
accuracy for training and validation using a custom CNN model.

Transfer Learning using VGG16 Model

For the second part, I used the VGG16 model with imagenet weights without the top layer and a custom denser layers at the end. Similar to the previous step, I used dropout layers and image augmentation. The exact command line for training this model is:

TrainCNN.py --cnnArch VGG16 --classMode Categorical --optimizer Adam --learningRate 1e-5 --imageSize 224 --numEpochs 30 --batchSize 25 --dropout --augmentation --augMultiplier 3

The CNN model is given below:

---------------------------------------------------------------
Model: "VGG16"
---------------------------------------------------------------   
Layer (type)                 Output Shape              Param #
===============================================================  
vgg16 (Model)                (None, 7, 7, 512)         14714688
flatten (Flatten)            (None, 25088)             0
dense (Dense)                (None, 512)               12845568
dense_1 (Dense)              (None, 256)               131328
dense_3 (Dense)              (None, 2)                 514
===============================================================
Total params: 27,692,104
Trainable params: 12,977,410
Non-trainable params: 14,714,688
---------------------------------------------------------------

The above model was trained on the same dataset as the custom model above and it achieved an accuracy of 98% after 11 epochs. Clearly, this model is far more efficient and more accurate then the custom designed model.

Cross-entropy loss for training and validation and the classification accuracy
for training and validation using a transfer learning from VGG16 model.

References:

  1. https://www.kaggle.com/c/dogs-vs-cats
  2. https://github.com/saurabhg17/ExploringDeepLearning

String Selection Widget for Qt5

Some time back, I developed a data entry application in Qt5. One of the requirements was to let the user select a single string from a predefined list of string. I developed a custom widget called SStringSelector for this purpose. SStringSelector has two views: display and selection. The display view presents the currently selected string (blank if no string is selected), and a push button. To select a string, the user clicks on the button which presents the user with the selection dialog. The selection dialog consists of a list of string in an QListWidget and the user can select one of them by double-clicking a string. If the list of strings are long, the user can filter them using a filter QLineEdit present above the QListWidget.

SStringSelector is distributed as a part of QtUtils repository hosted on Github. The SStringSelector widget is really simple to use. Simple add the SStringSelector.h and SStringSelector.cpp files in your project and add an instance of SStringSelector in the layout of your app.

Below are some screenshots of the widget under Windows:

The SStringSelector Widget.
Selection Dialog of the SStringSelector Widget.
Filtering Strings in the Selection Dialog.

Color Picker Widget for Qt5

Qt5 support standard dialogs such as QFileDialog, QFontDialog, and QColorDialog, however, it does not provide a color picker to allow a user to pick a color. Recently, I need a color picker for one of my projects and I implemented a simple color picker widget.

SColorPicker is available from Github as a part of QtUtils repository. To use SColorPicker, add the header and cpp files directly in your project. Then, simply add an instance of SColorPicker in a layout. SColorPicker will appear as 16×16 pixels colored square in the layout. If you need a different size, change it in the SColorPicker's constructor. When a user double-clicks on the colored square, the system’s color dialog will appear allowing the user to choose a color. The selected color can be obtained from color() function or by connecting to colorPicked() signal.

Below are the screenshots of the SColorPicker_Demo and system color dialog present to the user on Windows 10 computer.