|
|
@@ -1,586 +0,0 @@
|
|
|
-\documentclass[9pt,technote]{IEEEtran}
|
|
|
-\usepackage{amssymb, amsmath} % needed for math
|
|
|
-\usepackage{hyperref} % links im text
|
|
|
-\usepackage{parskip}
|
|
|
-\usepackage[pdftex,final]{graphicx}
|
|
|
-\usepackage{csquotes}
|
|
|
-\usepackage{braket}
|
|
|
-\usepackage{booktabs}
|
|
|
-\usepackage{multirow}
|
|
|
-\usepackage{pgfplots}
|
|
|
-\usepackage{ wasysym }
|
|
|
-\usepackage[noadjust]{cite}
|
|
|
-\usepackage[nameinlink,noabbrev]{cleveref} % has to be after hyperref, ntheorem, amsthm
|
|
|
-\usepackage[binary-units]{siunitx}
|
|
|
-\sisetup{per-mode=fraction,binary-units=true}
|
|
|
-\DeclareSIUnit\pixel{px}
|
|
|
-\usepackage{glossaries}
|
|
|
-\loadglsentries[main]{glossary}
|
|
|
-\makeglossaries
|
|
|
-
|
|
|
-\title{On-line Recognition of Handwritten Mathematical Symbols}
|
|
|
-\author{Martin Thoma, Kevin Kilgour, Sebastian St{\"u}ker and Alexander Waibel}
|
|
|
-
|
|
|
-\hypersetup{
|
|
|
- pdfauthor = {Martin Thoma, Kevin Kilgour, Sebastian St{\"u}ker and Alexander Waibel},
|
|
|
- pdfkeywords = {Mathematics,Symbols,recognition},
|
|
|
- pdftitle = {On-line Recognition of Handwritten Mathematical Symbols}
|
|
|
-}
|
|
|
-\include{variables}
|
|
|
-
|
|
|
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
-% Begin document %
|
|
|
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
-\begin{document}
|
|
|
-\maketitle
|
|
|
-\begin{abstract}
|
|
|
-
|
|
|
-The automatic recognition of single handwritten symbols has three main
|
|
|
-applications. The first application is to support users who know how a symbol
|
|
|
-looks like, but not what its name is such as $\saturn$. The second application
|
|
|
-is providing the necessary commands for professional publishing in books or on
|
|
|
-websites, e.g. in form of \LaTeX{} commands, as MathML, or as code points. The
|
|
|
-third application of single symbol classifiers is in form of a building block
|
|
|
-for formula recognition.
|
|
|
-
|
|
|
-This paper presents a system
|
|
|
-which uses the pen trajectory to classify handwritten symbols. Five
|
|
|
-preprocessing steps, one data multiplication algorithm, five features and five
|
|
|
-variants for multilayer Perceptron training were evaluated using $\num{166898}$
|
|
|
-recordings which were collected with two crowdsourcing projects. The evaluation
|
|
|
-results of these 21~experiments were used to create an optimized recognizer
|
|
|
-which has a TOP-1 error of less than $\SI{17.5}{\percent}$ and a TOP-3 error of
|
|
|
-$\SI{4.0}{\percent}$. This is a relative improvement of $\SI{18.5}{\percent}$
|
|
|
-for the TOP-1 error and $\SI{29.7}{\percent}$ for the TOP-3 error compared to
|
|
|
-the baseline system.
|
|
|
-\end{abstract}
|
|
|
-
|
|
|
-\section{Introduction}
|
|
|
-On-line recognition makes use of the pen trajectory. This means the data is
|
|
|
-given as groups of sequences of tuples $(x, y, t) \in \mathbb{R}^3$, where each
|
|
|
-group represents a stroke, $(x, y)$ is the position of the pen on a canvas and
|
|
|
-$t$ is the time.
|
|
|
-
|
|
|
-On-line data was used to classify handwritten natural language text in many
|
|
|
-different variants. For example, the NPen++ system classified cursive
|
|
|
-handwriting into English words by using hidden Markov models and neural
|
|
|
-networks\cite{Manke1995}.
|
|
|
-
|
|
|
-% One handwritten symbol in the described format is called a
|
|
|
-% \textit{recording}. One approach to classify recordings into symbol classes
|
|
|
-% assigns a probability to each class given the data. The classifier can be
|
|
|
-% evaluated by using recordings which were classified by humans and were not used
|
|
|
-% to train the classifier. The set of those recordings is called \textit{test
|
|
|
-% set}. The TOP-$n$ error is defined as the fraction of the symbols where
|
|
|
-% the correct class was not within the top $n$ classes of the highest
|
|
|
-% probability.
|
|
|
-
|
|
|
-Several systems for mathematical symbol recognition with on-line data have been
|
|
|
-described so far~\cite{Kosmala98,Mouchere2013}, but no standard test set
|
|
|
-existed to compare the results of different classifiers. The used symbols
|
|
|
-differed in all papers. This is unfortunate as the choice of symbols is crucial
|
|
|
-for the TOP-$n$ error. For example, the symbols $o$, $O$, $\circ$ and $0$ are
|
|
|
-very similar and systems which know all those classes will certainly have a
|
|
|
-higher TOP-$n$ error than systems which only accept one of them. But not only
|
|
|
-the classes differed, also the used data to train and test had to be collected
|
|
|
-by each author again.
|
|
|
-
|
|
|
-Daniel Kirsch describes in~\cite{Kirsch} a system called Detexify which uses
|
|
|
-time warping to classify on-line handwritten symbols and reports a TOP-3 error
|
|
|
-of less than $\SI{10}{\percent}$ for a set of $\num{100}$~symbols. He did also
|
|
|
-recently publish his data on \url{https://github.com/kirel/detexify-data},
|
|
|
-which was collected by a crowdsourcing approach via
|
|
|
-\url{http://detexify.kirelabs.org}. Those recordings as well as some recordings
|
|
|
-which were collected by a similar approach via \url{http://write-math.com} were
|
|
|
-used to train and evaluated different classifiers. A complete description of
|
|
|
-all involved software, data and experiments is given in~\cite{Thoma:2014}.
|
|
|
-
|
|
|
-In this paper we present a baseline system for the classification of on-line
|
|
|
-handwriting into $369$ classes of which some are very similar. An optimized
|
|
|
-classifier which has a $\SI{29.7}{\percent}$ relative improvement of the TOP-3
|
|
|
-error. This was achieved by using better features and layer-wise supervised
|
|
|
-pretraining. The absolute improvements compared to the baseline of those
|
|
|
-changes will also be shown.
|
|
|
-
|
|
|
-
|
|
|
-\section{Steps in Handwriting Recognition}
|
|
|
-The following steps are used for symbol classification:\nobreak
|
|
|
-\begin{enumerate}
|
|
|
- \item \textbf{Preprocessing}: Recorded data is never perfect. Devices have
|
|
|
- errors and people make mistakes while using the devices. To tackle
|
|
|
- these problems there are preprocessing algorithms to clean the data.
|
|
|
- The preprocessing algorithms can also remove unnecessary variations
|
|
|
- of the data that do not help in the classification process, but hide
|
|
|
- what is important. Having slightly different sizes of the same symbol
|
|
|
- is an example of such a variation. Four preprocessing algorithms that
|
|
|
- clean or normalize recordings are explained in
|
|
|
- \cref{sec:preprocessing}.
|
|
|
- \item \textbf{Data multiplication}: Learning algorithms need lots of data
|
|
|
- to learn internal parameters. If there is not enough data available,
|
|
|
- domain knowledge can be considered to create new artificial data from
|
|
|
- the original data. In the domain of on-line handwriting recognition,
|
|
|
- data can be multiplied by adding rotated variants.
|
|
|
- \item \textbf{Segmentation}: The task of formula recognition can eventually
|
|
|
- be reduced to the task of symbol recognition combined with symbol
|
|
|
- placement. Before symbol recognition can be done, the formula has
|
|
|
- to be segmented. As this paper is only about single-symbol
|
|
|
- recognition, this step will not be further discussed.
|
|
|
- \item \textbf{Feature computation}: A feature is high-level information
|
|
|
- derived from the raw data after preprocessing. Some systems like
|
|
|
- Detexify take the result of the preprocessing step, but many compute
|
|
|
- new features. Those features could be designed by a human engineer or
|
|
|
- learned. Non-raw data features can have the advantage that less
|
|
|
- training data is needed since the developer can use knowledge about
|
|
|
- handwriting to compute highly discriminative features. Various
|
|
|
- features are explained in \cref{sec:features}.
|
|
|
- \item \textbf{Feature enhancement}: Applying PCA, LDA, or
|
|
|
- feature standardization might change the features in ways that could
|
|
|
- improve the performance of learning algorithms.
|
|
|
-\end{enumerate}
|
|
|
-
|
|
|
-After these steps, we are faced with a classification learning task which
|
|
|
-consists of two parts:
|
|
|
-\begin{enumerate}
|
|
|
- \item \textbf{Learning} parameters for a given classifier.
|
|
|
- \item \textbf{Classifying} new recordings, sometimes called
|
|
|
- \textit{evaluation}. This should not be confused with the evaluation
|
|
|
- of the classification performance which is done for multiple
|
|
|
- topologies, preprocessing queues, and features in
|
|
|
- \Cref{ch:Evaluation}.
|
|
|
-\end{enumerate}
|
|
|
-
|
|
|
-The classification learning task can be solved with \glspl{MLP} if the number
|
|
|
-of input features is the same for every recording. There are many ways how to
|
|
|
-adjust \glspl{MLP} and how to adjust their training. Some of them are
|
|
|
-described in~\cref{sec:mlp-training}.
|
|
|
-
|
|
|
-
|
|
|
-\section{Data and Implementation}
|
|
|
-The combined data of Detexify and \href{http://write-math.com}{write-math.com}
|
|
|
-can be downloaded via \href{http://write-math.com/data}{write-math.com/data} as
|
|
|
-a compressed tar archive. It contains a list of $369$ symbols which are used in
|
|
|
-mathematical context. Each symbol has at least $50$ labeled examples, but most
|
|
|
-symbols have more than $200$ labeled examples and some have more than $2000$.
|
|
|
-In total, more than $\num{160000}$ labeled recordings were collected.
|
|
|
-
|
|
|
-Preprocessing and feature computation algorithms were implemented and are
|
|
|
-publicly available as open-source software in the Python package \texttt{hwrt}
|
|
|
-and \gls{MLP} algorithms are available in the Python package
|
|
|
-\texttt{nntoolkit}.
|
|
|
-
|
|
|
-
|
|
|
-\section{Algorithms}
|
|
|
-\subsection{Preprocessing}\label{sec:preprocessing}
|
|
|
-Preprocessing in symbol recognition is done to improve the quality and
|
|
|
-expressive power of the data. It should make follow-up tasks like segmentation
|
|
|
-and feature extraction easier, more effective or faster. It does so by resolving
|
|
|
-errors in the input data, reducing duplicate information and removing irrelevant
|
|
|
-information.
|
|
|
-
|
|
|
-Preprocessing algorithms fall into two groups: Normalization and noise
|
|
|
-reduction algorithms.
|
|
|
-
|
|
|
-A very important normalization algorithm in single-symbol recognition is
|
|
|
-\textit{scale-and-shift}~\cite{Thoma:2014}. It scales the recording so that
|
|
|
-its bounding box fits into a unit square. As the aspect ratio of a recording
|
|
|
-is almost never 1:1, only one dimension will fit exactly in the unit square.
|
|
|
-There are multiple ways how to shift the recording. For this paper, it was
|
|
|
-chosen to shift the bigger dimension to fit into the $[0,1] \times [0,1]$ unit
|
|
|
-square whereas the smaller dimension is centered in the $[-1,1] \times [-1,1]$
|
|
|
-square.
|
|
|
-
|
|
|
-Another normalization preprocessing algorithm is resampling. As the data points
|
|
|
-on the pen trajectory are generated asynchronously and with different
|
|
|
-time-resolutions depending on the used hardware and software, it is desirable
|
|
|
-to resample the recordings to have points spread equally in time for every
|
|
|
-recording. This was done by linear interpolation of the $(x,t)$ and $(y,t)$
|
|
|
-sequences and getting a fixed number of equally spaced points per stroke.
|
|
|
-
|
|
|
-\textit{Connect strokes} is a noise reduction algorithm. It happens sometimes
|
|
|
-that the hardware detects that the user lifted the pen where the user certainly
|
|
|
-didn't do so. This can be detected by measuring the Euclidean distance between
|
|
|
-the end of one stroke and the beginning of the next stroke. If this distance is
|
|
|
-below a threshold, then the strokes are connected.
|
|
|
-
|
|
|
-Due to a limited resolution of the recording device and due to erratic
|
|
|
-handwriting, the pen trajectory might not be smooth. One way to smooth is
|
|
|
-calculating a weighted average and replacing points by the weighted average of
|
|
|
-their coordinate and their neighbors coordinates. Another way to do smoothing
|
|
|
-would be to reduce the number of points with the Douglas-Peucker algorithm to
|
|
|
-the most relevant ones and then interpolate the stroke between those points.
|
|
|
-The Douglas-Peucker stroke simplification algorithm is usually used in
|
|
|
-cartography to simplify the shape of roads. It works recursively to find a
|
|
|
-subset of points of a stroke that is simpler and still similar to the original
|
|
|
-shape. The algorithm adds the first and the last point $p_1$ and $p_n$ of a
|
|
|
-stroke to the simplified set of points $S$. Then it searches the point $p_i$ in
|
|
|
-between that has maximum distance from the line $p_1 p_n$. If this
|
|
|
-distance is above a threshold $\varepsilon$, the point $p_i$ is added to $S$.
|
|
|
-Then the algorithm gets applied to $p_1 p_i$ and $p_i p_n$ recursively. It is
|
|
|
-described as \enquote{Algorithm 1} in~\cite{Visvalingam1990}.
|
|
|
-
|
|
|
-\subsection{Features}\label{sec:features}
|
|
|
-Features can be \textit{global}, that means calculated for the complete
|
|
|
-recording or complete strokes. Other features are calculated for single points
|
|
|
-on the pen trajectory and are called \textit{local}.
|
|
|
-
|
|
|
-Global features are the \textit{number of strokes} in a recording, the
|
|
|
-\textit{aspect ratio} of a recordings bounding box or the
|
|
|
-\textit{ink} being used for a recording. The ink feature gets calculated by
|
|
|
-measuring the length of all strokes combined. The re-curvature, which was
|
|
|
-introduced in~\cite{Huang06}, is defined as
|
|
|
-\[\text{re-curvature}(stroke) := \frac{\text{height}(stroke)}{\text{length}(stroke)}\]
|
|
|
-and a stroke-global feature.
|
|
|
-
|
|
|
-The simplest local feature is the coordinate of the point itself. Speed,
|
|
|
-curvature and a local small-resolution bitmap around the point, which was
|
|
|
-introduced by Manke, Finke and Waibel in~\cite{Manke1995}, are other local
|
|
|
-features.
|
|
|
-
|
|
|
-\subsection{Multilayer Perceptrons}\label{sec:mlp-training}
|
|
|
-\Glspl{MLP} are explained in detail in~\cite{Mitchell97}. They can have
|
|
|
-different numbers of hidden layers, the number of neurons per layer and the
|
|
|
-activation functions can be varied. The learning algorithm is parameterized by
|
|
|
-the learning rate $\eta \in (0, \infty)$, the momentum $\alpha \in [0, \infty)$
|
|
|
-and the number of epochs.
|
|
|
-
|
|
|
-The topology of \glspl{MLP} will be denoted in the following by separating the
|
|
|
-number of neurons per layer with colons. For example, the notation
|
|
|
-$160{:}500{:}500{:}500{:}369$ means that the input layer gets 160~features,
|
|
|
-there are three hidden layers with 500~neurons per layer and one output layer
|
|
|
-with 369~neurons.
|
|
|
-
|
|
|
-\glspl{MLP} training can be executed in various different ways, for example
|
|
|
-with \gls{SLP}. In case of a \gls{MLP} with the topology
|
|
|
-$160{:}500{:}500{:}500{:}369$, \gls{SLP} works as follows: At first a \gls{MLP}
|
|
|
-with one hidden layer ($160{:}500{:}369$) is trained. Then the output layer is
|
|
|
-discarded, a new hidden layer and a new output layer is added and it is trained
|
|
|
-again, resulting in a $160{:}500{:}500{:}369$ \gls{MLP}. The output layer is
|
|
|
-discarded again, a new hidden layer is added and a new output layer is added
|
|
|
-and the training is executed again.
|
|
|
-
|
|
|
-Denoising auto-encoders are another way of pretraining. An
|
|
|
-\textit{auto-encoder} is a neural network that is trained to restore its input.
|
|
|
-This means the number of input neurons is equal to the number of output
|
|
|
-neurons. The weights define an \textit{encoding} of the input that allows
|
|
|
-restoring the input. As the neural network finds the encoding by itself, it is
|
|
|
-called auto-encoder. If the hidden layer is smaller than the input layer, it
|
|
|
-can be used for dimensionality reduction~\cite{Hinton1989}. If only one hidden
|
|
|
-layer with linear activation functions is used, then the hidden layer contains
|
|
|
-the principal components after training~\cite{Duda2001}.
|
|
|
-
|
|
|
-Denoising auto-encoders are a variant introduced in~\cite{Vincent2008} that
|
|
|
-is more robust to partial corruption of the input features. It is trained to
|
|
|
-get robust by adding noise to the input features.
|
|
|
-
|
|
|
-There are multiple ways how noise can be added. Gaussian noise and
|
|
|
-randomly masking elements with zero are two possibilities. \cite{Deeplearning-Denoising-AE}
|
|
|
-describes how such a denoising auto-encoder with masking noise can be
|
|
|
-implemented. The \texttt{corruption} is the probability of a feature being
|
|
|
-masked.
|
|
|
-
|
|
|
-\section{Evaluation}\label{ch:Evaluation}
|
|
|
-In order to evaluate the effect of different preprocessing algorithms, features
|
|
|
-and adjustments in the \gls{MLP} training and topology, the following baseline
|
|
|
-system was used:
|
|
|
-
|
|
|
-Scale the recording to fit into a unit square while keeping the aspect ratio,
|
|
|
-shift it into $[-1,1] \times [-1,1]$ as described in \cref{sec:preprocessing},
|
|
|
-resample it with linear interpolation to get 20~points per stroke, spaced
|
|
|
-evenly in time. Take the first 4~strokes with 20~points per stroke and
|
|
|
-2~coordinates per point as features, resulting in 160~features which is equal
|
|
|
-to the number of input neurons. If a recording has less than 4~strokes, the
|
|
|
-remaining features were filled with zeroes.
|
|
|
-
|
|
|
-All experiments were evaluated with four baseline systems $B_i$, $i \in \Set{1,
|
|
|
-2, 3, 4}$, where $i$ is the number of hidden layers as different topologies
|
|
|
-could have a severe influence on the effect of new features or preprocessing
|
|
|
-steps. Each hidden layer in all evaluated systems has $500$ neurons.
|
|
|
-
|
|
|
-Each \gls{MLP} was trained with a learning rate of $\eta = 0.1$ and a momentum
|
|
|
-of $\alpha = 0.1$. The activation function of every neuron in a hidden layer is
|
|
|
-the sigmoid function $\text{sig}(x) := \frac{1}{1+e^{-x}}$. The neurons in the
|
|
|
-output layer use the softmax function. For every experiment, exactly one part
|
|
|
-of the baseline systems was changed.
|
|
|
-
|
|
|
-\subsection{Random Weight Initialization}
|
|
|
-The neural networks in all experiments got initialized with a small random
|
|
|
-weight
|
|
|
-
|
|
|
-\[w_{i,j} \sim U(-4 \cdot \sqrt{\frac{6}{n_l + n_{l+1}}}, 4 \cdot \sqrt{\frac{6}{n_l + n_{l+1}}})\]
|
|
|
-
|
|
|
-where $w_{i,j}$ is the weight between the neurons $i$ and $j$, $l$ is the layer
|
|
|
-of neuron $i$, and $n_i$ is the number of neurons in layer $i$. This random
|
|
|
-initialization was suggested on
|
|
|
-\cite{deeplearningweights} and is done to break symmetry.
|
|
|
-
|
|
|
-This might lead to different error rates for the same systems just because the
|
|
|
-initialization was different.
|
|
|
-
|
|
|
-In order to get an impression of the magnitude of the influence on the different
|
|
|
-topologies and error rates the baseline models were trained 5 times with
|
|
|
-random initializations.
|
|
|
-\Cref{table:baseline-systems-random-initializations-summary}
|
|
|
-shows a summary of the results. The more hidden layers are used, the more do
|
|
|
-the results vary between different random weight initializations.
|
|
|
-
|
|
|
-\begin{table}[h]
|
|
|
- \centering
|
|
|
- \begin{tabular}{crrr|rrr} %chktex 44
|
|
|
- \toprule
|
|
|
- \multirow{3}{*}{System} & \multicolumn{6}{c}{Classification error}\\
|
|
|
- \cmidrule(l){2-7}
|
|
|
- & \multicolumn{3}{c}{TOP-1} & \multicolumn{3}{c}{TOP-3}\\
|
|
|
- & min & max & range & min & max & range\\\midrule
|
|
|
- $B_1$ & $\SI{23.08}{\percent}$ & $\SI{23.44}{\percent}$ & $\SI{0.36}{\percent}$ & $\SI{6.67}{\percent}$ & $\SI{6.80}{\percent}$ & $\SI{0.13}{\percent}$ \\
|
|
|
- $B_2$ & \underline{$\SI{21.45}{\percent}$} & \underline{$\SI{21.83}{\percent}$}& $\SI{0.38}{\percent}$ & $\SI{5.68}{\percent}$ & \underline{$\SI{5.75}{\percent}$} & $\SI{0.07}{\percent}$\\
|
|
|
- $B_3$ & $\SI{21.54}{\percent}$ & $\SI{22.28}{\percent}$ & $\SI{0.74}{\percent}$ & \underline{$\SI{5.50}{\percent}$} & $\SI{5.82}{\percent}$ & $\SI{0.32}{\percent}$\\
|
|
|
- $B_4$ & $\SI{23.19}{\percent}$ & $\SI{24.84}{\percent}$ & $\SI{1.65}{\percent}$ & $\SI{5.98}{\percent}$ & $\SI{6.44}{\percent}$ & $\SI{0.46}{\percent}$\\
|
|
|
- \bottomrule
|
|
|
- \end{tabular}
|
|
|
- \caption{The systems $B_1$ -- $B_4$ were randomly initialized, trained
|
|
|
- and evaluated 5~times to estimate the influence of random weight
|
|
|
- initialization.}
|
|
|
-\label{table:baseline-systems-random-initializations-summary}
|
|
|
-\end{table}
|
|
|
-
|
|
|
-\subsection{Connect strokes}
|
|
|
-In order to solve the problem of interrupted strokes, pairs of strokes
|
|
|
-can be connected with stroke connect algorithm. The idea is that for
|
|
|
-a pair of consecutively drawn strokes $s_{i}, s_{i+1}$ the last point $s_i$ is
|
|
|
-close to the first point of $s_{i+1}$ if a stroke was accidentally split
|
|
|
-into two strokes.
|
|
|
-
|
|
|
-$\SI{59}{\percent}$ of all stroke pair distances in the collected data are
|
|
|
-between $\SI{30}{\pixel}$ and $\SI{150}{\pixel}$. Hence the stroke connect
|
|
|
-algorithm was tried with $\SI{5}{\pixel}$, $\SI{10}{\pixel}$ and
|
|
|
-$\SI{20}{\pixel}$.
|
|
|
-All models TOP-3 error improved with a threshold of $\theta = \SI{10}{\pixel}$
|
|
|
-by at least $\SI{0.17}{\percent}$, except $B_4$ which improved only by
|
|
|
-$\SI{0.01}{\percent}$ which could be a result of random weight initialization.
|
|
|
-
|
|
|
-\subsection{Douglas-Peucker Smoothing}
|
|
|
-The Douglas-Peucker algorithm can be used to find
|
|
|
-points that are more relevant for the overall shape of a recording. After that,
|
|
|
-an interpolation can be done. If the interpolation is a cubic spline
|
|
|
-interpolation, this makes the recording smooth.
|
|
|
-
|
|
|
-The Douglas-Peucker algorithm was applied with a threshold of $\varepsilon =
|
|
|
-0.05$, $\varepsilon = 0.1$ and $\varepsilon = 0.2$ after scaling and shifting,
|
|
|
-but before resampling. The interpolation in the resampling step was done
|
|
|
-linearly and with cubic splines in two experiments. The recording was scaled
|
|
|
-and shifted again after the interpolation because the bounding box might have
|
|
|
-changed.
|
|
|
-
|
|
|
-The result of the application of the Douglas-Peucker smoothing with $\varepsilon
|
|
|
-> 0.05$ was a high rise of the TOP-1 and TOP-3 error for all models $B_i$.
|
|
|
-This means that the simplification process removes some relevant information and
|
|
|
-does not --- as it was expected --- remove only noise. For $\varepsilon = 0.05$
|
|
|
-with linear interpolation some models TOP-1 error improved, but the
|
|
|
-changes were small. It could be an effect of random weight initialization.
|
|
|
-However, cubic spline interpolation made all systems perform more than
|
|
|
-$\SI{1.7}{\percent}$ worse for TOP-1 and TOP-3 error.
|
|
|
-
|
|
|
-The lower the value of $\varepsilon$, the less does the recording change after
|
|
|
-this preprocessing step. As it was applied after scaling the recording such that
|
|
|
-the biggest dimension of the recording (width or height) is $1$, a value of
|
|
|
-$\varepsilon = 0.05$ means that a point has to move at least $\SI{5}{\percent}$
|
|
|
-of the biggest dimension.
|
|
|
-
|
|
|
-\subsection{Global Features}
|
|
|
-Single global features were added one at a time to the baseline systems. Those
|
|
|
-features were re-curvature $\text{re-curvature}(stroke) = \frac{\text{height}(stroke)}{\text{length}(stroke)}$
|
|
|
-as described in \cite{Huang06}, the ink feature which is the summed length
|
|
|
-of all strokes, the stroke count, the aspect ratio and the stroke center points
|
|
|
-for the first four strokes. The stroke center point feature improved the system
|
|
|
-$B_1$ by $\SI{0.27}{\percent}$ for the TOP-3 error and system $B_3$ for the
|
|
|
-TOP-1 error by $\SI{0.74}{\percent}$, but all other systems and error measures
|
|
|
-either got worse or did not improve much.
|
|
|
-
|
|
|
-The other global features did improve the systems $B_1 -- B_3$, but not $B_4$.
|
|
|
-The highest improvement was achieved with the re-curvature feature. It
|
|
|
-improved the systems $B_1 -- B_4$ by more than $\SI{0.6}{\percent}$ TOP-1 error.
|
|
|
-
|
|
|
-
|
|
|
-\subsection{Data Multiplication}
|
|
|
-Data multiplication can be used to make the model invariant to transformations.
|
|
|
-However, this idea seems not to work well in the domain of on-line handwritten
|
|
|
-mathematical symbols. It was tried to triple the data by adding a rotated
|
|
|
-version that is rotated 3 degrees to the left and another one that is rotated
|
|
|
-3 degrees to the right around the center of mass. This data multiplication
|
|
|
-made all classifiers for most error measures perform worse by more than
|
|
|
-$\SI{2}{\percent}$ for the TOP-1 error.
|
|
|
-
|
|
|
-\subsection{Pretraining}\label{subsec:pretraining-evaluation}
|
|
|
-Pretraining is a technique used to improve the training of \glspl{MLP} with
|
|
|
-multiple hidden layers.
|
|
|
-
|
|
|
-\Cref{fig:training-and-test-error-for-different-topologies-pretraining} shows
|
|
|
-the evolution of the TOP-1 error over 1000~epochs with supervised
|
|
|
-layer-wise pretraining and without pretraining. It clearly shows that this
|
|
|
-kind of pretraining improves the classification performance by $\SI{1.6}{\percent}$
|
|
|
-for the TOP-1 error and $\SI{1.0}{\percent}$ for the TOP-3 error.
|
|
|
-
|
|
|
-\begin{figure}[htb]
|
|
|
- \centering
|
|
|
- \input{figures/errors-by-epoch-pretraining/errors-by-epoch-pretraining.tex}
|
|
|
- \caption{Training- and test error by number of trained epochs for different
|
|
|
- topologies with \gls{SLP}. The plot shows
|
|
|
- that all pretrained systems performed much better than the systems
|
|
|
- without pretraining. All plotted systems did not improve
|
|
|
- with more epochs of training.}
|
|
|
-\label{fig:training-and-test-error-for-different-topologies-pretraining}
|
|
|
-\end{figure}
|
|
|
-
|
|
|
-Pretraining with denoising auto-encoder lead to the much worse results listed in
|
|
|
-\cref{table:pretraining-denoising-auto-encoder}. The first layer used a $\tanh$
|
|
|
-activation function. Every layer was trained for $1000$ epochs and the
|
|
|
-\gls{MSE} loss function. A learning-rate of $\eta = 0.001$, a corruption of
|
|
|
-$0.3$ and a $L_2$ regularization of $\lambda = 10^{-4}$ were chosen. This
|
|
|
-pretraining setup made all systems with all error measures perform much worse.
|
|
|
-
|
|
|
-\begin{table}[tb]
|
|
|
- \centering
|
|
|
- \begin{tabular}{lrrrr}
|
|
|
- \toprule
|
|
|
- \multirow{2}{*}{System} & \multicolumn{4}{c}{Classification error}\\
|
|
|
- \cmidrule(l){2-5}
|
|
|
- & TOP-1 & change & TOP-3 & change \\\midrule
|
|
|
- $B_{1,p}$ & $\SI{23.75}{\percent}$ & $\SI{+0.41}{\percent}$ & $\SI{7.19}{\percent}$ & $\SI{+0.39}{\percent}$\\
|
|
|
- $B_{2,p}$ & \underline{$\SI{22.76}{\percent}$} & $\SI{+1.25}{\percent}$ & $\SI{6.38}{\percent}$ & $\SI{+0.63}{\percent}$\\
|
|
|
- $B_{3,p}$ & $\SI{23.10}{\percent}$ & $\SI{+1.17}{\percent}$ & \underline{$\SI{6.14}{\percent}$} & $\SI{+0.40}{\percent}$\\
|
|
|
- $B_{4,p}$ & $\SI{25.59}{\percent}$ & $\SI{+1.71}{\percent}$ & $\SI{6.99}{\percent}$ & $\SI{+0.87}{\percent}$\\
|
|
|
- \bottomrule
|
|
|
- \end{tabular}
|
|
|
- \caption{Systems with denoising auto-encoder pretraining compared to pure
|
|
|
- gradient descent. The pretrained systems clearly performed worse.}
|
|
|
-\label{table:pretraining-denoising-auto-encoder}
|
|
|
-\end{table}
|
|
|
-
|
|
|
-\subsection{Optimized Recognizer}
|
|
|
-All preprocessing steps and features that were useful were combined to
|
|
|
-create a recognizer that should perform best.
|
|
|
-
|
|
|
-All models were much better than everything that was tried before. The results
|
|
|
-of this experiment show that single-symbol recognition with
|
|
|
-\totalClassesAnalyzed{} classes and usual touch devices and the mouse can be
|
|
|
-done with a TOP1 error rate of $\SI{18.56}{\percent}$ and a TOP3 error of
|
|
|
-$\SI{4.11}{\percent}$. This was
|
|
|
-achieved by a \gls{MLP} with a $167{:}500{:}500{:}\totalClassesAnalyzed{}$ topology.
|
|
|
-
|
|
|
-It used an algorithm to connect strokes of which the ends were less than
|
|
|
-$\SI{10}{\pixel}$ away, scaled each recording to a unit square and shifted this
|
|
|
-unit square to $(0,0)$. After that, a linear resampling step was applied to the
|
|
|
-first 4 strokes to resample them to 20 points each. All other strokes were
|
|
|
-discarded.
|
|
|
-
|
|
|
-The 167 features were
|
|
|
-
|
|
|
-\begin{itemize}
|
|
|
- \item the first 4 strokes with 20 points per stroke resulting in 160
|
|
|
- features,
|
|
|
- \item the re-curvature for the first 4 strokes,
|
|
|
- \item the ink,
|
|
|
- \item the number of strokes and
|
|
|
- \item the aspect ratio
|
|
|
-\end{itemize}
|
|
|
-
|
|
|
-\Gls{SLP} was applied with $\num{1000}$ epochs per layer, a
|
|
|
-learning rate of $\eta=0.1$ and a momentum of $\alpha=0.1$. After that, the
|
|
|
-complete model was trained again for $1000$ epochs with standard mini-batch
|
|
|
-gradient descent.
|
|
|
-
|
|
|
-After the models $B_{1,c}$ -- $B_{4,c}$ were trained the first $1000$ epochs,
|
|
|
-they were trained again for $1000$ epochs with a learning rate of $\eta = 0.05$.
|
|
|
-\Cref{table:complex-recognizer-systems-evaluation} shows that
|
|
|
-this improved the classifiers again.
|
|
|
-
|
|
|
-\begin{table}[htb]
|
|
|
- \centering
|
|
|
- \begin{tabular}{lrrrr}
|
|
|
- \toprule
|
|
|
- \multirow{2}{*}{System} & \multicolumn{4}{c}{Classification error}\\
|
|
|
- \cmidrule(l){2-5}
|
|
|
- & TOP1 & change & TOP3 & change\\\midrule
|
|
|
- $B_{1,c}$ & $\SI{20.96}{\percent}$ & $\SI{-2.38}{\percent}$ & $\SI{5.24}{\percent}$ & $\SI{-1.56}{\percent}$\\
|
|
|
- $B_{2,c}$ & $\SI{18.26}{\percent}$ & $\SI{-3.25}{\percent}$ & $\SI{4.07}{\percent}$ & $\SI{-1.68}{\percent}$\\
|
|
|
- $B_{3,c}$ & \underline{$\SI{18.19}{\percent}$} & $\SI{-3.74}{\percent}$ & \underline{$\SI{4.06}{\percent}$} & $\SI{-1.68}{\percent}$\\
|
|
|
- $B_{4,c}$ & $\SI{18.57}{\percent}$ & $\SI{-5.31}{\percent}$ & $\SI{4.25}{\percent}$ & $\SI{-1.87}{\percent}$\\\midrule
|
|
|
- $B_{1,c}'$ & $\SI{19.33}{\percent}$ & $\SI{-1.63}{\percent}$ & $\SI{4.78}{\percent}$ & $\SI{-0.46}{\percent}$ \\
|
|
|
- $B_{2,c}'$ & \underline{$\SI{17.52}{\percent}$} & $\SI{-0.74}{\percent}$ & \underline{$\SI{4.04}{\percent}$} & $\SI{-0.03}{\percent}$\\
|
|
|
- $B_{3,c}'$ & $\SI{17.65}{\percent}$ & $\SI{-0.54}{\percent}$ & $\SI{4.07}{\percent}$ & $\SI{+0.01}{\percent}$\\
|
|
|
- $B_{4,c}'$ & $\SI{17.82}{\percent}$ & $\SI{-0.75}{\percent}$ & $\SI{4.26}{\percent}$ & $\SI{+0.01}{\percent}$\\
|
|
|
- \bottomrule
|
|
|
- \end{tabular}
|
|
|
- \caption{Error rates of the optimized recognizer systems. The systems
|
|
|
- $B_{i,c}'$ were trained another $1000$ epochs with a learning rate
|
|
|
- of $\eta=0.05$. The value of the column \enquote{change} of the
|
|
|
- systems $B_{i,c}'$ is relative to $B_{i,c}$.}
|
|
|
-\label{table:complex-recognizer-systems-evaluation}
|
|
|
-\end{table}
|
|
|
-
|
|
|
-
|
|
|
-\section{Discussion}
|
|
|
-Four baseline recognition systems were adjusted in many experiments and their
|
|
|
-recognition capabilities were compared in order to build a recognition system
|
|
|
-that can recognize 396 mathematical symbols with low error rates as well as to
|
|
|
-evaluate which preprocessing steps and features help to improve the recognition
|
|
|
-rate.
|
|
|
-
|
|
|
-All recognition systems were trained and evaluated with
|
|
|
-$\num{\totalCollectedRecordings{}}$ recordings for \totalClassesAnalyzed{}
|
|
|
-symbols. These recordings were collected by two crowdsourcing projects
|
|
|
-(\href{http://detexify.kirelabs.org/classify.html}{Detexify} and
|
|
|
-\href{write-math.com}{write-math.com}) and created with various devices. While
|
|
|
-some recordings were created with standard touch devices such as tablets and
|
|
|
-smartphones, others were created with the mouse.
|
|
|
-
|
|
|
-\Glspl{MLP} were used for the classification task. Four baseline systems with
|
|
|
-different numbers of hidden layers were used, as the number of hidden layer
|
|
|
-influences the capabilities and problems of \glspl{MLP}.
|
|
|
-
|
|
|
-All baseline systems used the same preprocessing queue. The recordings were
|
|
|
-scaled to fit into a unit square, shifted to $(0,0)$, resampled with linear
|
|
|
-interpolation so that every stroke had exactly 20~points which are spread
|
|
|
-equidistant in time. The 80~($x,y$) coordinates of the first 4~strokes were used
|
|
|
-to get exactly $160$ input features for every recording. The baseline system
|
|
|
-$B_2$ has a TOP-3 error of $\SI{5.75}{\percent}$.
|
|
|
-
|
|
|
-Adding two slightly rotated variants for each recording and hence tripling the
|
|
|
-training set made the systems $B_3$ and $B_4$ perform much worse, but improved
|
|
|
-the performance of the smaller systems.
|
|
|
-
|
|
|
-The global features re-curvature, ink, stoke count and aspect ratio improved the
|
|
|
-systems $B_1$--$B_3$, whereas the stroke center point feature made $B_2$ perform
|
|
|
-worse.
|
|
|
-
|
|
|
-Denoising auto-encoders were evaluated as one way
|
|
|
-to use pretraining, but by this the error rate increased notably. However,
|
|
|
-supervised layer-wise pretraining improved the performance decidedly.
|
|
|
-
|
|
|
-The stroke connect algorithm was added to the preprocessing steps of the
|
|
|
-baseline system as well as the re-curvature feature, the ink feature, the number
|
|
|
-of strokes and the aspect ratio. The training setup of the baseline system was
|
|
|
-changed to supervised layer-wise pretraining and the resulting model was trained
|
|
|
-with a lower learning rate again. This optimized recognizer $B_{2,c}'$ had a TOP-3
|
|
|
-error of $\SI{4.04}{\percent}$. This means that the TOP-3 error dropped by over
|
|
|
-$\SI{1.7}{\percent}$ in comparison to the baseline system $B_2$.
|
|
|
-
|
|
|
-A TOP-3 error of $\SI{4.04}{\percent}$ makes the system usable for symbol lookup.
|
|
|
-It could also be used as a starting point for the development of a
|
|
|
-multiple-symbol classifier.
|
|
|
-
|
|
|
-The aim of this work was to develop a symbol recognition system which is easy
|
|
|
-to use, fast and has high recognition rates as well as evaluating ideas for
|
|
|
-single symbol classifiers. Some of those goals were reached. The recognition
|
|
|
-system $B_{2,c}'$ evaluates new recordings in a fraction of a second and has
|
|
|
-acceptable recognition rates.
|
|
|
-
|
|
|
-% Many algorithms were evaluated.
|
|
|
-% However, there are still many other algorithms which could be evaluated and, at
|
|
|
-% the time of this work, the best classifier $B_{2,c}'$ is only available
|
|
|
-% through the Python package \texttt{hwrt}. It is planned to add an web version
|
|
|
-% of that classifier online.
|
|
|
-
|
|
|
-\bibliographystyle{IEEEtranSA}
|
|
|
-\bibliography{write-math-ba-paper}
|
|
|
-\end{document}
|
Martin Thoma
