latex-articles/else2019/tex/article.tex

\documentclass{article}


\usepackage{arxiv}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{lipsum}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{kbordermatrix}
\usepackage[
defernumbers=true,
backend=bibtex,
% style=authoryear,
style=numeric,
sorting=none
]{biblatex} % Use the bibtex backend with the authoryear citation style (which resembles APA)

\addbibresource{bibliografie.bib} 

\usepackage[section]{placeins}

\usepackage{caption}
\DeclareCaptionFormat{citation}{%
	\ifx\captioncitation\relax\relax\else
	\captioncitation\par
	\fi
	#1#2#3\par}
\newcommand*\setcaptioncitation[1]{\def\captioncitation{\textit{Sursa:}~#1}}
\let\captioncitation\relax

\captionsetup{format=citation,justification=centering,margin=40pt,font=small,labelfont=bf}

\usepackage{graphicx}
\graphicspath{ {images/} }

\title{MACHINE LEARNING BASED METHODS USED FOR IMPROVING SCHOLAR PERFORMANCE}


\author{
	Radu BONCEA \\
	ICI Bucuresti\\
	\texttt{radu@rotld.ro} \\
	\And
	Ionut PETRE \\
	ICI Bucuresti\\
	\texttt{ionut.petre@rotld.ro} \\
	\AND
	Victor VEVERA \\
	ICI Bucuresti\\
	\texttt{victor.vevera@ici.ro}
	\And
	Alexandru GHEORGHIȚĂ \\
	ICI Bucuresti\\
	\texttt{alex.gheorghita@rotld.ro}
}

\begin{document}
	\maketitle
	
	\begin{abstract}
		In their 1968 study \textit{"The Teaching-Learning Paradox: A Comparative Analysis of College Teaching Methods"}, Robert Dubin and Thomas Taveggia found no evidence to indicate any basis for preferring one teaching method over another as measured by the performance of students on course examinations. The conclusion is based on systematic reanalysis of the data of almost 100 comparative studies of different college teaching methods. The teaching-learning process is virtually a black box in which the teaching methods do not influence scholar performance. So rather than focusing on teaching methods, we propose a method for improving scholar performance by a continuous and intelligent monitoring and assessing process of daily knowledge gains. Recent developments in machine learning and data analysis, allow us the use of techniques for unveiling the strengths and weaknesses in the learning process.  Our proposed solution is “micro-assessing” each student and after each course, by sending a 5 minutes-long mini test on their smartphones, collects the result and send it for analysis to a dedicated platform. When the system has sufficient data, it can personalize the tests for each student with focus on areas the student is lacking. It can make recommendations, to teachers, students, school and competent authorities at local or national level. The solution is not a replacement for classical examinations, but it augments the learning experience through interactive and personalized quizzes. The teacher also has a better view over students knowledge, thus he can do better assessments overall.  Student abnormal \textit{deviations} could be detected much faster, while competent authorities could assess the impact of their decisions in near-realtime.
	\end{abstract}
	
	
	% keywords can be removed
	\keywords{scholar performance \and assessment model \and machine learning \and knowledge gain \and RLO \and IAMA}
	
	
	\section{Introduction}
	
		\paragraph{} 
			A study conducted in 1968 concluded that the teaching method has no impact on final exam performance \cite{rdtct1968}, a principle know as \textit{Teaching-Learning Paradox}. No matter the quality of teacher or the program, regardless of teaching methods and despite the circumstances before them, the students learned and they demonstrated their learning consistently across time. In his book "Why Don’t Students Like School", Daniel Willingham  provides a useful model for how the mind retains factual information (see Figure  ~\ref{fig:model_of_mind}) and points out that factual information is not sufficient for learning, but is often the basis for real learning, since in order to apply concepts and make creative evaluations, we must have a database of knowledge from which to pull and make connections \cite{danielw}. Thus, to achieve long-term memory, two things need to happen: 
		\begin{quote}
			\textit{1) said data must be subject to an intense level of attention while it is readily available in working memory, and... \\
			2) content previously stored in long-term memory must be pulled up and matched with the new data in order for it to permanently \emph{stick}} \cite{andrewneuendorf}....
		\end{quote} 
		\begin{figure}[ht]
			\centering
			\includegraphics[scale=0.7]{model_of_mind}
			\caption{Daniel Willingham's model of mind. }
			\label{fig:model_of_mind}
		\end{figure}
		\paragraph{} 
			In this article we are proposing new methods that are addressing the above two challenges and improve  scholar performance through means of demonstrating knowledge and understanding, by using an \textbf{I}terative (daily base) and \textbf{A}daptive \textbf{M}onitoring and \textbf{A}ssessing (IAMA) process that relies on state-of-art technologies which uncover the power of \textit{Data Intelligence} \cite{Drazdilova2010, Romero:2010:EDM:1922269.1922270, Wang:2009:DMC:1822686}.  One such method implies a process of \textit{micro-assessing} students right after each course using profiled Reusable Learning Objects (RLO) for assessment \cite{Neven:2002:RLO:641007.641067, JoDI89, SiqSean2003}. The assessment result is input data for generating and consolidating a student profile with focus on strengths and weaknesses. Data collected can also be used to monitor student performance and to alert and react to deviations. \emph{IAMA} could also be seen as a game of words such as the \emph{subreddit} question-and-answer \emph{iAMA} - \textit{\textbf{i}nteractive \textbf{A}sk \textbf{M}e \textbf{A}nything}.
		\paragraph{}
			It should be noted that the whole process is automated. The teacher allocates 5 minutes for each hour of course to assess students, by sending to their mobile electronic tablet, mini tests with 10 questions to be answered. Students receive the test and upon completion or expiring the time allocated, the applications installed on their mobile interfaces send the test results to a central platform (cloud based, data intensive), where they are stored. The system will process the results (compute scores, make inferences) and create and consolidate a knowledge-centered profile for each student. The profile is then used to recognize areas the student is lacking in knowledge and to provide with the input for future assessment adjustments and fine tuning.
	
	\section{IAMA Building Blocks}
		
		\subsection{Assessment Reusable Learning Object}
			\paragraph{}
				RLOs  are conceptualized as accessible, reusable, interoperable, and adaptable learning resources to facilitate developmental cost savings. Therefore, RLOs need to be designed as independent learning units that are free from context as well as links with external resources \cite{AJET3072}.
				Characteristics of RLOs:
				\begin{itemize}
					\item \textit{digital asset} - stored, retrieved, modified by electronic means, available online 24/7;
					\item \textit{self-contained} - loosely coupled, each learning object can be taken independently;
					\item \textit{reusable} – a single RLO may be used in multiple contexts for multiple purposes;
					\item \textit{searchable} - easy to find learning material; each RLO is tagged with metadata (see LOM subsection \ref{lom});
					\item \textit{atomicity} -  RLOs are ~5-15 minutes long, distinct, small \emph{units} of knowledge information;
					\item \textit{flexible} – easy to update and change;
					\item \textit{standardized} – adopt the same organizational structure from an enterprise architecture perspective;
					\item \textit{aggregability} – learning objects can be grouped into a larger collection of content, including traditional course structures;
					\item \textit{interoperability} – blend into Learning Management Systems (e.g. WebCT Vista, Moodle);
					\item \textit{digital adaptiblity} - RLOs are suited to address a new type of learner – “\emph{Net-generation learner}” adapted to multi-tasking and digital technologies;
					\item \textit{student-focus} - enhance student-centered learning.
				\end{itemize}
			\paragraph{}
				An assessment RLO (aRLO) is a RLO that is in essence a question associated with a set of possible answers and a set of correct answers and has additional properties:
			\begin{itemize}
				\item \textit{mutability} - given a aRLO and a specific algorithm, we can obtain a set of distinctive aRLOs. A very simple algorithm would be to randomly sort the answers. A more complex  algorithm is to make an inference on the question as seen in Figure \ref{fig:rlo_mutability}.
				\item \textit{affiliation} - aRLOs are associated with one ore more knowledge domains (parts of the curriculum) and memory domains (e.g. recall, identify, recognize, recount, relate,etc).
			\end{itemize}
			\begin{figure}[ht]
				\centering
				\includegraphics[scale=0.4]{rlo_mutability}
				\caption{Inference based mutability of RLOs}
				\label{fig:rlo_mutability}
			\end{figure}
			
	
		\subsection{Learning Object Metadata}
			\label{lom}
			\paragraph{}
				Learning Object Metadata (LOM) \cite{1032843} is a data model, used to describe a l\emph{earning object} and similar digital resources used to support learning. It is based on 1484.12.1 IEEE multi-part standard and has been developed with the purpose to facilitate search, evaluation, acquisition, and use of learning objects, for instance by learners or instructors or automated software processes. LOM also facilitates the sharing and exchange of learning objects, by enabling the development of catalogs and inventories while taking into account the diversity of cultural and lingual contexts in which the learning objects and their metadata are reused. It is a natural choice to describe RLOs.\\
				The LOM conceptual data schema has a hierarchical tree structure composed of the following nine categories \cite{LCampbell}:
				\begin{enumerate}
					\item \textit{General}: information that describes the learning object as a whole.
					\item \textit{Lifecycle}: history and current status of the learning object and those who have contributed to its creation.
					\item \textit{Meta-metadata}: information about the metadata describing the learning object, as opposed to the learning object itself.
					\item \textit{Technical}: technical requirements and characteristics of the learning object.
					\item \textit{Educational}: educational and pedagogic characteristics of the learning object..
					\item \textit{Rights}: intellectual property rights and conditions of use of the learning object.
					\item \textit{Relation}: relationship between the learning object and other related objects.
					\item \textit{Annotation}: comments on the educational use of the learning objects, including when and by whom the comments were created.
					\item \textit{Classification}: classification schemes used to describe different characteristics of the learning object.
				\end{enumerate}
			\paragraph{}
			As seen in Figure \ref{fig:lom_schema}, the LOM schema is relatively extensive  and, in most cases, application profiles based on the standard generally restrict the elements used, designate certain elements as mandatory or optional, specify vocabulary usage and interpretation, and add organization or community specific classification schemes. And in some cases, the LOM schema must be extended with new properties. In our case, the aRLO should extend the \emph{general} category, so it includes the two possible/correct answer sets.
			
		\begin{figure}[ht]
			\centering
			\includegraphics[scale=0.5]{lom_schema}
			\caption{LOM conceptual data schema}
			\label{fig:lom_schema}
		\end{figure}
	
		\subsection{Semantic Repository}
			\paragraph{}
				A semantic repository is a  database management systems, capable of handling structured data, taking into consideration their semantics and is perfectly suited to storing RLOs \cite{Ochoa}. We can use Resource Description Framework (RDF) \cite{532408} to express RLO metadata following the IEEE LOM standard. A perfect candidate for storing and handling RDF structures is Apache Jena \cite{apachejenaurl}, a free and open source Java framework for building semantic web and linked data applications that offers:
				\begin{itemize}
					\item RDF API to create and read RDF graphs and serialize the triples using popular formats such as RDF/XML or Turtle;
					\item graph-oriented query language ARQ, which is a SPARQL 1.1 compliant engine. ARQ supports remote federated queries and free text search via Lucene;
					\item a fast persistent triple store that stores directly to disk (TDB);
					\item ontology API working with models, RDFS and the Web Ontology Language (OWL) to add extra semantics to RDF data;
					\item inference API wrapping functionality around built-in OWL and RDFS reasoners.
				\end{itemize}
				
		\subsection{Machine Learning Models and Frameworks}
			\paragraph{}
				Monitoring student performance over time based on small iterative tests implies that we should \emph{feed} the  test results into an algorithm that outputs a score or a list of scores for a domain or a set of knowledge domains.  There are several machine learning models that are well suited for doing classification and regression \cite{Mandeep}:
				\begin{enumerate}
					\item \textit{Naive Bayes Classifier} is a probabilistic classifier based on Bayes Theorem with an assumption of independence among predictors and particularly useful for very large data sets. It is simple to implement and is known to outperform even highly sophisticated classification methods.
					\item \textit{Logistic Regression} is the appropriate regression predictive analysis to conduct when the dependent variable (the outcome) is dichotomous (binary).
					\item \textit{Decision Trees} builds classification or regression models in the form of a tree structure, by breaking down a data set into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision nodes and leaf nodes.
					\item \textit{Random Forests} are an ensemble learning method for classification, regression and other tasks, that operate by constructing a multitude of decision trees. This model is very accurate, runs efficiently on large data sets and has a very useful feature: it gives estimates of what variables are important in the classification.
					\item \textit{Neural Network} consists of units (neurons), arranged in layers, which convert an input vector into some output. Each unit takes an input, applies a (often nonlinear) function to it and then passes the output on to the next layer. Generally the networks are defined to be feed-forward: a unit feeds its output to all the units on the next layer, but there is no feedback to the previous layer. Weightings are applied to the signals passing from one unit to another, and it is these weightings which are tuned in the training phase to adapt a neural network to the particular problem at hand.
					\item \textit{Nearest Neighbor} is a supervised, non-parametric, classification and regression algorithm, where the input consists of the \emph{k} closest training examples in the feature space.
				\end{enumerate}
			\paragraph{}
				These algorithms can be implemented using low level frameworks, such as SciPy \cite{scipy} and Scikit Learn \cite{scikit}, highly used, simple, yet efficient Python libraries built on \emph{NumPy}, \emph{pandas}, and \emph{matplotlib}.  For implementing neural networks, specifically deep learning models, we should use higher level frameworks such as TensorFlow \cite{tensorflow}, CNTK \cite{cntk}, PyTorch \cite{pytorch} or Keras \cite{keras}.
			
	\section{The Solution}
		\paragraph{}
			Let  $\mathcal{S}\epsilon\mathcal{R}^{m \times n}$ annotate the associative matrix of computed scores for all \emph{m} students and all \emph{n} aRLOs, where $\mathcal{R}$  is the closed interval $[0,1] =\{x \epsilon \mathbb{R}; 0\leq x \leq 1\}$. As mentioned before an aRLO is an assessment reusable object that has attached to it a set of possible answers and a set of correct answers and a difficulty level $\delta \epsilon (0,1]$ where $\delta=1$ means the most difficult level. Each element $\sigma_i$ in the matrix $\mathcal{S}$ is calculated as $\sigma_i = \gamma_i \times \delta_i$ where $\gamma_i$ is the grade for the corresponding aRLO (students get maximum grade 1 if they get all correct answers right ) and $\delta_i$ is the difficulty level of the aRLO \cite{BONCEA2018PRO}.\\
			Let $\mathcal{D}\epsilon\{0,1\}^{n \times p}$ annotate the matrix of mappings between knowledge domains and aRLOs and $\mathcal{M}\epsilon\{0,1\}^{n \times q}$ be  the matrix of mappings between memory domains and aRLOs, where \emph{n} is the number of aRLOs, \emph{p} is the number of knowledge domains and \emph{q} is the number of memory domains. 
		\paragraph{}
			Then, we calculate the scores for associated knowledge domains $\mathcal{S}_{\mathcal{D}}$ with the formula $(\mathcal{S} \mathcal{D} )_{ij} = \frac{\sum_{k=1}^n \mathcal{S}_{i,k} \times \mathcal{D}_{k,j}}{N} =  \mathcal{S} \bigotimes \mathcal{D}$, where \emph{N} is the number of domain associations of each aRLO (the count of \textit{1}s).\\ 
			 Similarly, we calculate the scores for associated  memory domains as $\mathcal{M}$: $\mathcal{S}_{\mathcal{M}} = \mathcal{S} \bigotimes \mathcal{M}$.
			
			For exemplification, let's consider the following matrixes: the grade scores \emph{Gs} with \emph{m} questions and \emph{n} students, the knowledge domains associated matrix $\mathcal{D}$ and the memory domains matrix $\mathcal{M}$,
			\renewcommand{\kbldelim}{(}% Left delimiter
			\renewcommand{\kbrdelim}{)}% Right delimiter
			\[
			\text{\emph{Gs}} = \kbordermatrix{
				& S_1 & S_2 & S_3 & ... & S_n \\
				\gamma_1 & 1 & 1 & 0 & ... & 1 \\
				\gamma_2 & 1 & 0 & 1 & ... & 1 \\
				\gamma_3 & 0 & 1 & 1 & ... & 1 \\
				 ... & ... & ... & ... & ... & ... \\
				\gamma_m & 1 & 0 & 1 & ... & 1
			}
			\]
			\[
			\mathcal{D} = \kbordermatrix{
				& \gamma_1 & \gamma_2 & \gamma_3 & ... & \gamma_m \\
				\delta_1 & 1 & 0 & 0 & ... & 0 \\
				\delta_2 & 0 & 1 & 0 & ... & 0 \\
				\delta_3 & 0 & 0 & 1 & ... & 1 \\
				... & ... & ... & ... & ... & ... \\
				\delta_p & 1 & 0 & 1 & ... & 0
			}
			\]
			
			\[
			\mathcal{M} = \kbordermatrix{
				& \gamma_1 & \gamma_2 & \gamma_3 & ... & \gamma_m \\
				\mu_1 & 0 & 0 & 0 & ... & 1 \\
				\mu_2 & 0 & 1 & 0 & ... & 0 \\
				\mu_3 & 1 & 0 & 1 & ... & 0 \\
				... & ... & ... & ... & ... & ... \\
				\mu_q & 0 & 0 & 1 & ... & 0
			}
			\]
			
			and the difficulty levels \emph{Diff} 
			\[
			\text{\emph{Diff}} = \kbordermatrix{
				& \gamma_1 & \gamma_2 & \gamma_3 & ... & \gamma_m \\
			    & 0.8 & 0.6 & 0.6 & ... & 1 \\
			}
			\]
			we calculate the score matrix $\mathcal{S}$, the knowledge domain scores $\mathcal{S}_{\mathcal{D}}$ and the memory domain scores $\mathcal{S}_{\mathcal{M}}$
			\[
				\mathcal{S} =
				\begin{bmatrix}
					 1 \times 0.8 = 0.8 & 1 \times 0.8 = 0.8  & 0 \times 0.8 = 0 & ... &  1 \times 0.8 = 0.8\\
					 1 \times 0.6 = 0.6 & 0 \times 0.6 = 0  & 1 \times 0.6 = 0.6 & ... &  1 \times 0.6 = 0.6\\
					 0 \times 0.6 = 0 & 1 \times 0.6 = 0.6  & 1 \times 0.6 = 0.6 & ... &  1 \times 0.6 = 0.6\\
					 ... & ...  & ... & ... &  ...\\
					 1 \times 1 = 1 & 1 \times 1 = 1  & 1 \times 1 = 1 & ... &  1 \times 1 = 1\\
				\end{bmatrix}
				= 
				\begin{bmatrix}
				0.8 &  0.8  &  0 & ... &  0.8\\
				0.6 & 0 & 0.6 & ... &  0.6\\
				0 & 0.6  & 0.6 & ... &  0.6\\
				... & ...  & ... & ... &  ...\\
				1 & 1  & 1 & ... &  1\\
				\end{bmatrix}
			\]
			
			\[
				\mathcal{S}_{\mathcal{D}} = \mathcal{S} \bigotimes \mathcal{D} = 
				\begin{bmatrix}
				0.8 &  0.8  &  0 & ... &  0.8\\
				0.6 & 0 & 0.6 & ... &  0.6\\
				0 & 0.6  & 0.6 & ... &  0.6\\
				... & ...  & ... & ... &  ...\\
				1 & 1  & 1 & ... &  1
				\end{bmatrix}
				\bigotimes
				\begin{bmatrix}
					 1 & 0 & 0 & ... & 0 \\
					 0 & 1 & 0 & ... & 0 \\
					 0 & 0 & 1 & ... & 1 \\
					... & ... & ... & ... & ... \\
					 1 & 0 & 1 & ... & 0
				\end{bmatrix}
				=
				\begin{bmatrix}
				\frac{0.8+0.8}{2} & \frac{0.8}{1} & \frac{0.8}{1}  & ... &  0 \\
				\frac{0.12}{2} & 0 & \frac{0.12}{2} & ... & \frac{0.6}{1} \\
				0.6 & 0.6 & 0.6 & ... & 0.6 \\
				... & ... & ... & ... & ... \\
				1 & 1 & 1 & ... & 1
				\end{bmatrix}
		\]
		\[
		=
		\kbordermatrix{
			& \delta_1 & \delta_2 & \delta_3 & ... & \delta_p \\
		S_1 & 0.8 & 0.8 & 0.8  & ... &  0 \\
		S_2 &0.6 & 0 & 0.6 & ... & 0.6 \\
		S_3 &0.6 & 0.6 & 0.6 & ... & 0.6 \\
		... & ... & ... & ... & ... \\
		S_n &1 & 1 & 1 & ... & 1
		}
		\]
		
		\[
		\mathcal{S}_{\mathcal{M}} = \mathcal{S} \bigotimes \mathcal{M} = 
		\begin{bmatrix}
		0.8 &  0.8  &  0 & ... &  0.8\\
		0.6 & 0 & 0.6 & ... &  0.6\\
		0 & 0.6  & 0.6 & ... &  0.6\\
		... & ...  & ... & ... &  ...\\
		1 & 1  & 1 & ... &  1
		\end{bmatrix}
		\bigotimes
		\begin{bmatrix}
		0 & 0 & 0 & ... & 1 \\
		0 & 1 & 0 & ... & 0 \\
		1 & 0 & 1 & ... & 0 \\
		... & ... & ... & ... & ... \\
		0 & 0 & 1 & ... & 0
		\end{bmatrix}
		=
		\]
			\[
		=
		\kbordermatrix{
			& \mu_1 & \mu_2 & \mu_3 & ... & \mu_q \\
			S_1 & 0 & 0.8 & 0.8  & ... &  0.8 \\
			S_2 &0.6 & 0 & 0.6 & ... & 0 \\
			S_3 &0.6 & 0.6 & 0.6 & ... & 0 \\
			... & ... & ... & ... & ... \\
			S_n &1 & 1 & 1 & ... & 1
		}
		\]
		
	\paragraph{}
		In our previous example we can conclude that the \emph{n}\textsuperscript{th} student has perfect score in each  domain of knowledge and each  domain of memory. The knowledge domain that is best demonstrated is $\delta_3$, while $\delta_p$ is the worst.
	\paragraph{}
		We then store the tensor  $\mathcal{T} = (\mathcal{S},\mathcal{S}_{\mathcal{D}},\mathcal{S}_{\mathcal{M}} )$ in a non-relational database like MongoDB or Cassandra and, after a certain number of iterations or mini tests, we can infer new relations using the machine learning models described before. For instance we could classify the students on knowledge levels (classification problem) or we can compute a score in the interval [0,1] (multivariate regression problem). The result could then be used to provide with profiled tests to student in domains they are less knowledgeable. 
		
	\paragraph{}
		Figure \ref{fig:architecture} shows a high level generic architecture of a IAMA solution, with three important aplication components:
		 \begin{enumerate}
		 	\item \textit{aRLO Management Component} is responsible with the development of aRLOs using LOM and RDF as semantic assets. aRLOs are stored and retreived from a semantic repository , such as Apache Jena. This component also handles the aRLO mutability, that is, given aRLO and inference rules, the component will generate a set of related aRLOs.
		 	\item \textit{Assessment Component} main functionlity is to generate profiled tests for students and handle the test results by storing them into a datastore.
		 	\item \textit{Data Analysis Component} uses machine learning models to generate the student profiles and have new insights.
		 \end{enumerate}
	 	The three components are well integrated in order to provide with business services that can be used by a large variety of actors (e.g. teachers, students, parent, local authorities,goverment), including the integration into larger application services ecosystem.
		
		\begin{figure}[ht]
			\centering
			\includegraphics[scale=0.5]{architecture}
			\caption{IAMA Reference Architecture}
			\label{fig:architecture}
		\end{figure}
		
	\section{Conclusions}
		\paragraph{}
		The solution addreses key challenges in teaching-learning process, by using data intelligence and machine learning in order to monitor the student performance over time. The IAMA model is based on an iterative and adaptive evaluation process that's using semantic and higly extensible reusable learning objects implemented on LOM and serialized and stored as RDF structures. The RLOs are associated with one or more knowledge domains (identifiable components of curriculum), one or more memory domains or long-term memory capabilities  (e.g. recall, recount, recognize, relate, identify) and a difficulty level. The test results are stored for future retrieval and processed using machine learning algorithms suited for solving classification and regression problems. The main objective of this process is to obtain a student's profile which is continuosly updated and improved. The student profile is then used to tune the assessment with focus on domains the student is less knowledgeable.
		
		\paragraph{}
		 Except for RLO creation process, in IAMA, all other processes should be fully automated. The system should provide with distinct tests to all students in the class, making sure there are no students completing the same test. Assessments should contain a certain number of RLOs that are associated with previous courses so that all memory domains are sufficiently addressed. The same requirement is for knowledge domains coverage.\\
		 But, the RLO creation could also be automated in some areas, though human supervision is a requirement that can't be overlooked. New RLOs can be proceduraly generated using inference rules or more simple algorithms such as permutations of possible answers. Proceduralaly generated RLOs inherit most of the properties from the parent RLO, including affiliation (domain associative matrixes).
		 
		\paragraph{}
		Besides the score, domain and memory matrixes, we can always add new features. For instance, we could additionally add the response time for each RLO or the sex, age and student's gender, or provide with data regarding the classroom, such as the number of students in class. And we can always verify which are the most important features, which features have greater impact on the classification or regression final result, by using \emph{Random Forests}. This process can be further extended by including features regarding the school, the city, the country. Thus, the results can be aggregated into critical information for better decisions by teachers, students, parents and competent authorities.\\
		It is the feature extensibility that requires our solution to store the assessment results and associated data into a schema-free and non-relational database.
	\nocite{*}
	\printbibliography[heading=bibintoc, title={Bibliografie}]
\end{document}