Reinforcement Learning (RL) model for self-improving chatbots, specifically targeting FAQ-type chatbots.

Elena Ricciardelli, Debmalya Biswas

Abstract. We present a Reinforcement Learning (RL) model for self-improving chatbots, specifically targeting FAQ-type chatbots. The model is not aimed at building a dialog system from scratch, but to leverage data from user conversations to improve chatbot performance. At the core of our approach is a score model, which is trained to score chatbot utterance-response tuples based on user feedback. The scores predicted by this model are used as rewards for the RL agent. Policy learning takes place offline, thanks to an user simulator which is fed with utterances from the FAQ-database. Policy learning is implemented using a Deep Q-Network (DQN) agent with epsilon-greedy exploration, which is tailored to effectively include fallback answers for out-of-scope questions. The potential of our approach is shown on a small case extracted from an enterprise chatbot. …

Manage licensing risks of Open Source Data Science projects

This is an extended article accompanying the presentation on “Open Source Enterprise AI/ML Governance” at Linux Foundation’s Open Compliance Summit, Dec 2020 (link) (pptx)

Image for post
Image for post
Open Source Software Licensing: based on the pic by Andrea Piacquadio from Pexels

Open Source Software Enterprise Adoption Trends

The best AI/ML software today from model development (scikit-learn, TensorFlow, PyTorch) to deployment (Kubeflow, Spark) is Open Source. According to Gartner, “more than 90% of companies rely on Open Source Software”. The below snapshot should give you an idea of the pervasiveness of Open Source Software (OSS) in the Enterprise.

Image for post
Image for post
OSS enterprise adoption trends

While OSS started as a cost cutting exercise, it has increasingly become the preferred platform for innovation.

However, this entails a big change for the (large) enterprise Compliance/legal teams, who in many cases had a strict ‘No Open Source’ policy so far. …

Reproducibility, Explainability, Bias & Accountability

This is an extended article accompanying the presentation on “Open Source Enterprise AI/ML Governance” at Linux Foundation’s Open Compliance Summit, Dec 2020 (link) (pptx)

For the last 4–5 years, we have been working hard towards implementing various AI/ML use-cases at our enterprises. We have been focusing on building the most performant models, and now that we have a few of them in production; it is time to move beyond model precision/recall rates to a more holistic Enterprise AI Governance framework that ensures that our AI adoption is in line with our organizational principles and policies.

By ‘enterprises’ in this article, I am NOT referring to Google, Facebook, or similar digitally native organisations. Rather, this article is focused on the many banks, telecoms, pharmas, FMCGs, etc. who have now realized the value of AI advancements; and are now racing towards adopting them in a very agile fashion. …

Develop personalized apps using a combination of Reinforcement Learning and NLP/Chatbots

Abstract. We present a Reinforcement Learning (RL) based approach to implement Recommender Systems. The results are based on a real-life Wellness app that is able to provide personalized health / activity related content to users in an interactive fashion. Unfortunately, current recommender systems are unable to adapt to continuously evolving features, e.g. user sentiment, and scenarios where the RL reward needs to computed based on multiple and unreliable feedback channels (e.g., sensors, wearables). …

HVAC optimization in factories for a sustainable future

Abstract. Heating, Ventilation and Air Conditioning (HVAC) units are responsible for maintaining the temperature and humidity settings in a building. Studies have shown that HVAC accounts for almost 50% energy consumption in a building and 10% of global electricity usage. HVAC optimization thus has the potential to contribute significantly towards our sustainability goals, reducing energy consumption and CO2 emissions. In this work, we explore ways to optimize the HVAC controls in factories. Unfortunately, this is a complex problem as it requires computing an optimal state considering multiple variable factors, e.g. the occupancy, manufacturing schedule, temperature requirements of operating machines, air flow dynamics within the building, external weather conditions, energy savings, etc. We present a Reinforcement Learning (RL) based energy optimization model that has been applied in our factories. We show that RL is a good fit as it is able to learn and adapt to multi-parameterized system dynamics in real-time. …


Do (will) we still need Data Scientists?

AutoML tools have been gaining traction for the last couple of years, both due to technological advancements and their potential to be leveraged by ‘Citizen Data Scientists’. Citizen Data Science, is an interesting (often controversial) aspect of Data Science (DS) that aims to automate the design of Machine Learning (ML)/Deep Learning (DL) models, making it more accessible to people without the specialized skills of a Data Scientist.

In this article, we will try to understand AutoML, its promise, what is possible today?, where AutoML fails (today)?, …

Privacy preserving NLP based on Entity Filtering and Searchable Encryption

Abstract. With chatbots gaining traction and their adoption growing in different verticals, e.g. Health, Banking, Dating; and users sharing more and more private information with chatbots — studies have started to highlight the privacy risks of chatbots. In this paper, we propose two privacy-preserving approaches for chatbot conversations. The first approach applies ‘entity’ based privacy filtering and transformation, and can be applied directly on the app (client) side. It however requires knowledge of the chatbot design to be enabled. We present a second scheme based on Searchable Encryption that is able to preserve user chat privacy, without requiring any knowledge of the chatbot design. …

Getting Started

3-tier Chatbot architecture integrating Search

This is an informal take on the technological choice that we are often faced with when designing Chatbots, i.e. should we build a Chatbot or a Natural Language Search (NLS), or a may be a mix of both? The primary motivation of both is to make enterprise data and applications (more) accessible to every company employee — to foster knowledge sharing and collaboration. With this objective, we explore two integration architectures:

  • Search enabled Chatbots
  • Conversational Search


Much has been said and written about Chatbots. However, most of this discussion is focused around Consumer facing bots — the multi-million dollar bot that will radically transform your company’s image and allow you to save a few million on the sidelines. It goes without saying that building such a singular bot also requires investment in the order of a few hundred thousand dollars. …

Student ‘notes’ personalization & marketplace

Image for post
Image for post
Photo by Anna Shvets from Pexels


Massive Open Online Courses (MOOCs) have been gaining traction over the last few years, with every major university launching their online channel and the advent of digital platforms, e.g. Coursera, Udemy, edX. This adoption has been further accelerated by the Coronavirus pandemic where many schools / universities have been forced to switch to the online medium. So the online delivery of lectures has moved from a niche market to the mainstream product used by billions of people worldwide.

Innovation in the online education field has so far focused on improving the lecture content and making it more engaging for students, e.g., using gaze tracking to assess student attention [1], and gamification techniques to improve the content interactiveness [2]. …

Need for an Open Source Scanning tool and the RFP template to select the most effective tool

In the previous article (link), we discussed ‘what is Open Source Software (OSS)?’, ‘why it is important for your enterprise?’, and ‘what are the operational risks posed by OSS?’

By this point, you have hopefully started using OSS extensively in your enterprises, and we will focus on the steps needed to effectively manage OSS, including setting up of an Open Source Program Office / Open Source Review Board (OSRB), Governance Framework and Scanning Tool.

Open Source Program Office (OSPO)

OSPO is the “central place where all open source activities are handled for consistent communication inside/outside the company.” — HP Open Source Governance (link).

It is an interdisciplinary team involving IT, Legal and Procurement with the following…


Debmalya Biswas

AI and Open Source Architect | Ex-Nokia, SAP, Oracle | 50+ Patents |

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store