Reinforcement Learning in web recommender systems

This time I found a pretty interesting form of representing a problem in the web to fit a reinforcement learning approach. Since the information in the Web is increasing day after day, it is necessary to provide systems that could recommend users related content of interest to them. Web content recommendation has been an active application area for information filtering, web mining and machine learning research. In this paper the authors exploit a way to enhance a reinforcement learning solution, that has been devised for web recommendations based on web usage data.

To model the problem as reinforcement learning, they use the analogy of a game in which the system is constantly trying to predict the next state of a user web browsing session, knowing his previous requests (visited pages), and the history of other users browsing sessions. The action is selecting a recommended page. Reward (Rs) is in function of the visited pages and the already recommended pages for each state. A state S’ is rewarded when the last page visited belongs to the recommended pages list.

One interesting aspect of this system, that actually relates to my thesis topic, is that this approach not only takes into account the urls of the web pages in order to record the web usage history, but it also relates those pages with concepts, providing semantics to this recommendations (i.e the system at the same time of recommending urls of web pages it also recommends concepts related to what the user is actually browsing). In this case, semantics provide a more interesting point of view for this recommendation system based on reinforcement learning, and the recommended pages for the user actually match concepts that he/she is browsing, providing timely information reducing the effort to browse about user interests.

Reference:
Nima Taghipour and Ahmad Kardan. 2008. A hybrid web recommender system based on Q-learning. In Proceedings of the 2008 ACM symposium on Applied computing (SAC ’08). ACM, New York, NY, USA, 1164-1168. DOI=10.1145/1363686.1363954 http://doi.acm.org/10.1145/1363686.1363954