Rasa is a framework for creating virtual assistants. More than a year ago, when I wanted to develop my NLP and ML skills further, I ran into Rasa. I was nothing but amazed by this technology and decided that I wanted to master it.

Now, in this blog post, I am going to explain the basics of Rasa and how you can build a virtual assistant from scratch without prior knowledge of chatbot development.

So, you might be wondering now, what is Rasa? Rasa is a conversational AI platform for personalized conversations at scale. There are different products that Rasa offers, but in this post, I am going to focus on Rasa Open Source.

Rasa Open Source supplies the building blocks for creating virtual assistants and it’s used to automate human-to-computer interactions. What does this mean? With Rasa, we are able to create a complete chatbot, with everything we need, and ready to be utilized by our users. Furthermore,  we’ll be able to do more than hold a conversation, we can integrate APIs, and connect to messaging channels.

And now, hands on. How can we start working with Rasa? The first step is to install it. For that, as Rasa is based on Python, we will need to create a virtual environment, and after activating it, install it.

# Create a virtual environment
> python3 - venv ./venv
# Activate the virtual environment
> source ./venv/bin/activate
# Install Rasa
> pip3 install -U --user pip && pip3 install rasa

Now, we have Rasa installed in our environment and we are ready to start a new project. Again, it’s really simple to do that:

> rasa init

After a few minutes – Rasa installs a lot of ML, NLP and NLU libraries – we will have our new Rasa project created and ready to use. Rasa will also ask us if we want to perform the first training, and after that, we will be able to talk with our chatbot.

And right after the training finishes, we can talk with our bot!

At this point, we can only have a small chit chat with our chatbot. But from here, we can further develop our assistant to be whatever we want it to be!

Before digging into the implementation of new chatbot features I will explain the basic commands that we need to know to work with Rasa and after that, the different concepts that Rasa defines and which are very important in order to succeed.

Basic Commands
We already know one of them: rasa init. With this command, we can start and initialize a new project. We kinda know another one, that we implicitly run through the creation process: rasa train. After we do some changes in our code base, we need to run it as we need to train our model to reflect the new dialogue and conversation.

Then we have different commands like rasa shell, which starts an interactive shell where we can talk with our chatbot; or rasa run, in case we only want to run our chatbot and access it from our custom interface. And the last basic commands I am going to mention in this blog post, even though there are more and you should check them out here https://rasa.com/docs/rasa/command-line-interface are rasa run actions. This command will start our custom actions server (I will explain what this is later in this blog post).

Rasa Open Source Flow
Now that we know how to initiate a project and how to run the basic commands, it’s time to know how Rasa works at a high level.

In the following diagram, we can see what a usual conversation workflow would look like.

We have our system up and running, and when a user wants to converse with our assistant, he or she would type a message through the connector modules of our choice (either our custom interface or Messenger, Telegram, etc.). When the message is received, the Natural Language Understanding piece will take care of understanding what the user wants to do from the written text – identify the intent from the user input, extract useful information, etc. – and after that, thanks to the Machine Learning-based dialogue management we can react to that intent and provide the correct next action and even can access a database, API, etc. to fetch the necessary information so our user will enjoy a personalized conversation. After that, we select the text response and send it to the connector module, so it can be displayed to the user.

Project Structure
Now, going back to the code. We had generated a project with Rasa, but we don’t know yet what it looks like. Moreover, each file has a concept behind it and we need to learn it in order to succeed in creating a chatbot.

This is the project structure after running rasa init.

.
├── actions
│   ├── __init__.py
│   └── actions.py
├── config.yml
├── credentials.yml
├── data
│   ├── nlu.yml
│   └── stories.yml
├── domain.yml
├── endpoints.yml
├── models
│   └── <timestamp>.tar.gz
└── tests
   └── test_stories.yml

We will now go into detail for each file and learn the concepts that Rasa defines and what is the responsibility of each one.

Configuration: config.yml
In this file, we will define the components and policies that our model will use to make predictions based on the data input. Here, we will also define the language our chatbot operates.

The following code is an example of a config.yml file.

recipe: default.v1
language: en

pipeline:
# will be selected by the Suggested Config feature

policies:
- name: MemoizationPolicy
- name: TEDPolicy
  max_history: 5
  epochs: 10

I am not going to explain every detail of this file, I would need an entire blog post to do so, but for now, as an introduction, it is enough to know that here we can define policies and more configurations to get the most out of our chatbot.

Credentials: credentials.yml

Rasa is more than a backend piece for our chatbot. It offers a smooth way to communicate and connect our chatbot with different front ends and chat platforms like Slack, Messenger or Telegram, or even our custom frontend app. To do so, we can easily configure everything in the credentials.yml file.

The following code is an example of this file, where we can see how we can connect our assistant with different chat and voice platforms.

rest:
#  # you don't need to provide anything here - this channel doesn't
#  # require any credentials


#facebook:
#  verify: "<verify>"
#  secret: "<your secret>"
#  page-access-token: "<your page access token>"

#slack:
#  slack_token: "<your slack token>"
#  slack_channel: "<the slack channel>"
#  slack_signing_secret: "<your slack signing secret>"

#socketio:
#  user_message_evt: <event name for user message>
#  bot_message_evt: <event name for bot messages>
#  session_persistence: <true/false>

#mattermost:
#  url: "https://<mattermost instance>/api/v4"
#  token: "<bot token>"
#  webhook_url: "<callback URL>"

# This entry is needed if you are using Rasa X. The entry represents credentials
# for the Rasa X "channel", i.e. Talk to your bot and Share with guest testers.
rasa:
  url: "http://localhost:5002/api"

Endpoints: endpoints.yml

We can also integrate our assistant with different external services or endpoints. For example, we can connect an API to query data, or to an external database to store or update data. For this, we can configure our external endpoints in the endpoints.yml file.

You can check what this file looks like here:

# This file contains the different endpoints your bot can use.

# Server where the models are pulled from.
# https://rasa.com/docs/rasa/model-storage#fetching-models-from-a-server

#models:
#  url: http://my-server.com/models/default_core@latest
#  wait_time_between_pulls:  10   # [optional](default: 100)

# Server which runs your custom actions.
# https://rasa.com/docs/rasa/custom-actions

action_endpoint:
  url: "http://localhost:5055/webhook"

# Tracker store which is used to store the conversations.
# By default the conversations are stored in memory.
# https://rasa.com/docs/rasa/tracker-stores

#tracker_store:
#    type: redis
#    url: <host of the redis instance, e.g. localhost>
#    port: <port of your redis instance, usually 6379>
#    db: <number of your database within redis, e.g. 0>
#    password: <password used for authentication>
#    use_ssl: <whether or not the communication is encrypted, default false>

#tracker_store:
#    type: mongod
#    url: <url to your mongo instance, e.g. mongodb://localhost:27017>
#    db: <name of the db within your mongo instance, e.g. rasa>
#    username: <username used for authentication>
#    password: <password used for authentication>

# Event broker which all conversation events should be streamed to.
# https://rasa.com/docs/rasa/event-brokers

#event_broker:
#  url: localhost
#  username: username
#  password: password
#  queue: queue

Domain: domain.yml

We are now coming to the part where we’ll see more concepts that are related to the conversation itself. Whenever we want to define something that concerns the universe where our assistant operates, we need to go to the domain.yml file. There we have different concepts like intents, responses, slots, etc. we can define. We will go into more detail in the following subsections.

Intents

The first thing we can define in the domain.yml file is intents. In this section of the file, we can write a list of the intents that are used in our NLU data and conversation training data. We can tell, for each intent, if there are entities involved with the option use_entities or if we want to ignore some entities for given intent, we can specify that with the ignore_entities option.

It is also possible to ignore an entity for all intents by setting the influence_conversation flag to false in the entity itself. But we will explain this in the Entities section.

The following code is an example of the intents sections of the domain.yml file.

intents:
  - greet:
      use_entities:
        - name
  - goodbye
  - affirm:
      ignore_entities:
  - name
  - deny
  - mood_great
  - mood_unhappy
  - bot_challenge
  - weather

Responses

Responses in Rasa are actions that send a message to the user without executing code or returning events. It is usually only text, but we can also include images or buttons. We can define the responses inside the domain.yml file, or we can create a separate one called responses.yml.

Each response has to start with utter_. For example, utter_greet or utter_bye.

responses:
  utter_greet:
  - text: "Hey! How are you?"

  utter_cheer_up:
  - text: "Here is something to cheer you up:"
    image: "https://i.imgur.com/nGF1K8f.jpg"

  utter_did_that_help:
  - text: "Did that help you?"

  utter_happy:
  - text: "Great, carry on!"

  utter_goodbye:
  - text: "Bye"

  utter_iamabot:
  - text: "I am a bot, powered by Rasa."

  utter_weather:
  - text: "Today's temperature in {city} is {temp}°C"

Entities

Any entity that can be extracted by any entity extractor from our NLU pipeline should be listed here. We can add them in the domain.yml file, or create multiple files for declaring them. The entities can be referenced by any file and from any file. By default, entities will influence action prediction, but as we stated above, we can ignore specific entities in the intents themselves, or if we don’t want the entity to influence any intent, we can do it in the entity itself with the influence_conversation flag set to false.

entities:
  - city

Slots

Slots are our bot’s memory. With slots, we can allow our assistant to store specific details in memory and use them later. We can store both user-provided information and information gathered from the outside world (for example, the result of an API call).

We define slots in the domain.yml slots section. And we define the name, type and how they should influence the assistant’s behavior.

Exactly the same as entities, slots can influence the conversation or not, and the predefined behaviour is the same as the entities one.

We have different slot types (text, boolean, categorical, list etc.) and we can even create our custom slot class and use it. And there are multiple ways to map our slots, but I won’t get into much detail here, you can check all the options here https://rasa.com/docs/rasa/domain#slot-mappings.

slots:
  city:
    type: rasa.shared.core.slots.AnySlot
    initial_value: null
    influence_conversation: false
    mappings:
    - entity: city
      type: from_entity

Forms

Forms are the best way to allow our assistant to collect information to do something. This is also called “slot filling”.

The first thing we need to do is to add the policy “Rule Policy” to our configuration.

policies:
- name: RulePolicy

Then we need to define a form in the forms section of our domain.yml file. We need to give it a name and provide a list of mandatory slots for the form.

forms:
  restaurant_form:
    required_slots:
        - cuisine
        - num_people

In order to be able to “use” the form, we need to activate it. When this happens, the form will ask the user for the information to fill the slots, one by one. Rasa will do so by looking in the responses for utter_ask_<form_name>_<slot_name> or utter_ask_<slot_name> if the first one isn’t found.

In order to activate a form, we need to add either a story or a rule (or both). There we can describe when the assistant should run the form.

The following piece of code shows an example of a rule where an intent will trigger the form activation:

rules:
- rule: Activate form
  steps:
  - intent: request_restaurant
  - action: restaurant_form
  - active_loop: restaurant_form

And, as we activate the form, we also have to deactivate it. The form will automatically deactivate itself once all required slots are filled. We can also describe with a rule the behaviour of our assistant to end a form. If we don’t add an applicable rule or story, the assistant will listen to the user’s next message after the form finishes. 

rules:
- rule: Submit form
  condition:
  # Condition that form is active.
  - active_loop: restaurant_form
  steps:
  # Form is deactivated
  - action: restaurant_form
  - active_loop: null
  - slot_was_set:
    - requested_slot: null
  # The actions we want to run when the form is submitted.
  - action: utter_submit
  - action: utter_slots_values

We can also validate the input of our form by implementing a Custom Action (we will talk about Custom Actions in the next section).

Custom Actions

A custom action can run any code we want, including API calls, database queries, setting a slot, etc. They are used to execute specific code or any backend integration.

Any custom action we want to use should be added to the actions section in our domain.yml file.

actions:
- action_weather_api

We can run our custom action server with the following command rasa run actions.

Training data: data/

We have three different ways of training our data. All of them should be added under the data/ directory. Rasa uses .yaml files to define all three types of training data.

NLU

Natural Language Understanding’s (NLU’s) objective is to extract structured data from our user communications. The user’s intention and any entities their message contains are often included in this. Our training data can be enriched with additional information, including regular expressions and lookup tables, to assist the model in identifying intents and entities.

Rasa NLU training data stores structured information about user messages. It normally includes the user’s intents and the message entities. 

• Entities are structured pieces of information inside a user message. We need to consider what information our assistant needs to meet our user goals when determining which entities to extract. There is a chance that the user will supply more details that we don’t require for any user goals, thus we don’t have to extract them as entities.

If we have a weather chatbot, and we want to fetch the weather information for the provided city, we will have to define a city entity that needs to be extracted from the user input. In the following example, we can see that we have defined a city entity, and for our intent weather, we have provided different examples of how our users ask for the weather forecast, specifying our entity city.

entities:
  - city

- intent: weather
  examples: |
    - what's the weather in [London](city)
    - i want to know the weather for today in [Barcelona](city)

• Synonyms map extracted entities to a value other than the literal text extracted. We can use synonyms when there are multiple ways users refer to the same thing.

nlu:
- synonym: credit
  examples: |
    - credit card account
    - credit account

• Lookup tables are lists of words used to generate case-insensitive regular expression patterns. We can use lookup tables to help extract entities which have a known set of possible values. We should keep our lookup tables as specific as possible.

nlu:
- lookup: country
  examples: |
    - Afghanistan
    - Albania
    - ...
    - Zambia
    - Zimbabwe

Stories

It’s a type of training data for our assistant to learn the dialog of conversations. Stories can be used to train models that are able to generalize to unseen conversation paths.

A story represents a conversation between a user and an assistant. Inputs are represented as intents and bot responses as actions.

stories:
- story: collect restaurant booking info  # name of the story
  steps:
  - intent: greet                         # message without entities
  - action: utter_ask_howcanhelp
  - intent: inform                        # message with entities
    entities:
    - location: "rome"
    - price: "cheap"
  - action: utter_on_it                  # action that the bot executes
  - action: utter_ask_cuisine
  - intent: inform
    entities:
    - cuisine: "spanish"
  - action: utter_ask_num_people

Rules

A rule describes small fragments of conversations that should always follow the same direction.

Rules are good for handling small, particular conversational patterns, but we shouldn’t misuse them since, unlike stories, they cannot be generalized to cover unforeseen conversational paths. In order to make our assistant reliable and capable of handling actual user behavior, we combine rules and stories.

When writing a rule, first, we need the “Rule Policy” in our configuration. 

policies:
- ... # Other policies
- name: RulePolicy

Then, we can start adding the rules under the “Rule” section in our training data.

rules:

- rule: Say `hello` whenever the user sends a message with intent `greet`
  steps:
  - intent: greet
  - action: utter_greet

Now you know all the necessary steps and concepts for building your assistance with Rasa from scratch. 🏆