Reinforcement Learning in web recommender systems

This time I found a pretty interesting form of representing a problem in the web to fit a reinforcement learning approach. Since the information in the Web is increasing day after day, it is necessary to provide systems that could recommend users related content of interest to them. Web content recommendation has been an active application area for information filtering, web mining and machine learning research. In this paper the authors exploit a way to enhance a reinforcement learning solution, that has been devised for web recommendations based on web usage data.

To model the problem as reinforcement learning, they use the analogy of a game in which the system is constantly trying to predict the next state of a user web browsing session, knowing his previous requests (visited pages), and the history of other users browsing sessions. The action is selecting a recommended page. Reward (Rs) is in function of the visited pages and the already recommended pages for each state. A state S’ is rewarded when the last page visited belongs to the recommended pages list.

One interesting aspect of this system, that actually relates to my thesis topic, is that this approach not only takes into account the urls of the web pages in order to record the web usage history, but it also relates those pages with concepts, providing semantics to this recommendations (i.e the system at the same time of recommending urls of web pages it also recommends concepts related to what the user is actually browsing). In this case, semantics provide a more interesting point of view for this recommendation system based on reinforcement learning, and the recommended pages for the user actually match concepts that he/she is browsing, providing timely information reducing the effort to browse about user interests.

Nima Taghipour and Ahmad Kardan. 2008. A hybrid web recommender system based on Q-learning. In Proceedings of the 2008 ACM symposium on Applied computing (SAC ’08). ACM, New York, NY, USA, 1164-1168. DOI=10.1145/1363686.1363954


Bayesian Network for Ontology Mapping

I found one interesting application of Bayesian Networks for the Semantic Web, and it’s aimed to help ontology mapping between two different Ontologies, i.e. determine how much Onto1:Concept1 is similar to Onto2:Concept2, thus trying to map two different concepts obtaining a value that corresponds to the degree of similarity.
The details of how it works are pretty interesting. First, the Ontologies are converted to Bayesian Networks using a framework called BayesOWL (references on the paper); the resulting Bayesian Network preserves the semantics of the original ontologies, and support ontology reasoning, within and across ontologies, using Bayesian inferences. This BayesOWL framework provides the methods that utilizes available probability constraints about classes and inter-class relations in constructing the the conditional probability tables of the network.
Prior probability distributions of uncertainty about concepts used for the framework, conditional distributions for relations between classes in the same ontology and joint probability distributions for semantic similarity between different concepts in different ontologies, are constructed based on machine learning of these probabilities using text classification techniques, associating a concept with a group of sample text documents called exemplars, retrieved from a search engine.

The main idea of this research is to provide a simple and efficient way of determine the semantic similarity of two concepts in distinct ontologies. This is an approach that I didn’t know, but it’s worth to keep in mind when doing research in Semantic Web technologies.

Pan, Rong, et. Al. A Bayesian Network Approach to Ontology Mapping. The Semantic Web – ISWC 2005. Lecture Notes in Computer Science. 2005 Springer Berlin / Heidelberg.