Skip to main content

Kalliope acting as a reminder

edit: updated on 02/17

Introduction

Yesterday, I wanted to add a feature to Kalliope (my always-on voice controlled personal assistant) : the ability to reminds me of something later on. I have a very selective memory and forget to do a lot of stuff, so when I think about something important, I can gently ask Kalliope to reminds me to do it (:

The main use case

The primary use case (there are other things I'd like to do, see the last paragraph about the todolist) is:

Be able to ask Kalliope to reminds me of something in X minutes.

This translate into these actions:

  • A neuron with an order like "remind me in 10 minutes to leave for my appointment"
  • Launching a command that will sleep for 10minutes
  • After the delay, the command should make kalliope remind me of my appointment

I didn't / couldn't do a neuron, seeing the dependencies with languages and a need for an async script (kalliope needs to be able to answer commands in the meantime), a neuron was too complex.

My solution

I ended up creating:

  • A synapse that would just fire a python script that would do all the work.
  • The order looks like "remind me {{query}}".
  • The neuron used is the community shell neuron by sending the query to the script.
  • A python script that receive the full query (including time to wait and the "thing" to remember
  • The script fire an "at" command to program the wake up
  • The command started by "at" will leverage Kalliope API to tell me what to remember.
  • A new neuron to make kalliope say dynamic text.

The repeat neuron

The idea of this neuron is really simple: You send it arguments, and it will send them back as a return value to be used in template or say_template command.

As you can see on the sample brain file, you can do things like "say hello to {{name}}" or what not :).

To install the module, as usual:

.. code:: bash

kalliope install --git-url https://github.com/bacardi55/kalliope-repeat.git

Then you can configure your brain as usual.

On synapse I created was this one:

.. code:: yaml

  - name: "Repeat-via-api"
    signals:
      - order: "api-repeat-cmd {{ query }}"
    neurons:
      - repeat:
          args:
            - query
          say_template:
            - "{{ query }}"

The idea of this one is just to repeat what is in query when an order is sent that starts with "api-repeat-cmd". It's easy enough as I'm sure i will never pronounce a thing like this.

So basically, any API call that will POST an order starting like this will make Kalliope read the text sent.

For example,

.. code:: bash

curl -i --user admin:secret -H "Content-Type: application/json" -X POST -d '{"order":"api-repeat-cmd Hello sir, how are you ?"}' http://kalliope:5000/order/

This command will make Kalliope say "Hello sir, how are you".

So now, I have a neuron than can say anytext needed via API.

So leveraging this neuron, my at command could easily send me the text I asked Kalliope to reminds me (eg: "leave for my appointment")

The python script

The script needs the "at" program to be installed to run

The script can be found here

If you look at the script, you will notice it is heavily dependent on the language (in my case French, sorry), so you can't just take it and use it, you'll have to update the text in it. I kept the code comments in English to help though

And with this, you can know ask Kalliope to remind you stuff (:

The remaining part to do

  • Add a reminder at a specific time (at support time like 9:30) edit Now available
  • Delete last/all reminders (via atq and atrm commands)

Keep track of the python script and the example brain file, these features will arrive :).

Kalliope MPD neuron

Introduction

I was thinking about starting music by voice recognition for a while now. Since I've managed to launch my tv show via kalliope and kodi (more on that later :)), I still didn't have a way to do same for my music.

I start looking into existing solution and really liked the idea behind mopidy, having an MPD server that also works with remote services like spotify. Meaning I could control not only my local music (as I would with a simple MPD server) but also my spotify library!

So to make this happen, I started created a simple (for now) neuron to manage an MPD server via Kalliope.

I keep the full installation / configuration I made (kalliope + mopidy) for another blog post though (:

Installation

As usual, use the following command:

.. code:: bash

kalliope install --git-url https://github.com/bacardi55/kalliope-mpd.git

Configuration

This neuron let you launch several action. Let's look at a default neuron for using it:

.. code:: yaml

  - name: "play-music"
    signals:
      - order: "start playlist {{query}}"
    neurons:
      - kalliopempd:
          mpd_action: "playlist"
          mpd_url: "xxx.xxx.xxx.xxx"
          mpd_port: "yyyy"
          mpd_random: 1
          args:
            - query

As you can see, there is an mpd_action arguments that let you choose what to do. As of now, the available commands are:

  • playlist (requires query parameters): run a playlist
  • playlist_spotify (specific to mopidy - requires query parameters): run a playlist
  • toggle_play: to toggle play/pause
  • search (requires query parameters): to search music and play it
  • play_next: play next song
  • play_previsous: play previous song
  • play_stop: Stop playing

The playlist_spotify action is a bit specific to mopidy as if you want to run a "top playlist" on spotify (like top tracks everywhere in the glob) you'll need to use this action. If you know the name of your playlist, you can use the default playlist action to launch it (even for spotify playlist)

The README will have the up to date list.

More example are available in the sample brain file

Conclusion

Now I can start/stop music by voice and simply switch song (prev/next) or play/pause the music when needed (: I am also very pleased on how kalliope works well with music running in the background, so that the next/prev/pause commands are really usefull :).

I'll be back with a post about my full kalliope setups and configuration later when everything is finalized!

Manage dualscreen with HiDPI

My new Dell XPS 13 has a HiDPI touchscreen and when I plug a 2nd display (not HiDPI), the resolution is big and ugly.

Solution for this is using the --scale option of xrandr!

My script example (in this case, laptop screen is at the right of my external screen):

xrandr --output eDP1 --primary --mode 3200x1800 --pos 3840x192 --rotate normal --output DP1-1 --mode 1920x1080 --pos 0x0 --rotate normal --scale 2x2

Kalliope remote update

Hello!

I've already talked about kalliope remote on this blog, my kalliope web JS page to fire synapse remotely. You can find the previous blog posts here.

I've made a quick update to it so that now you can ignore some synapse (and un-ignore them via the "ignored synapse" page).

Updated synapse page:

![Updated synapse page]({{ site.url }}/assets/kalliopeember2.png)

I still need to work on the parameter part so that this app can fire synapses with arguments.

Stay tuned!

Web scrapping Kalliope neuron

Introduction

One of the things I use Kalliopé for is reading information on web sites, eg: reading the latest news of the French sport site "léquipe" or other sites. For this, I'm using the RSS neuron that works quite well :)

Unfortunately, some sites have decided that RSS feeds are too "has been" and if people wants to see what's new, they either have to follow the site on Twitter or just come on the site everyday… I found this behavior quite distasteful as I'm still a heavy RSS users…

So in order to be able to ask Kalliopé to read information on line for me, even when no RSS feed is available, I created a simple web scraper neuron.

Installation

.. code:: bash

kalliope install --git-url https://github.com/bacardi55/kalliope-web-scraper.git

Configuration

Once installed, the neuron is pretty simple: you give it a URL, an html tag that contains the list of data you want Kalliopé to read. Then on this list you give Kalliopé what is the html tag for the title and the html for the description.

For example, on google news:

The main selector is : "div.top-stories-section div.section-content div.story" and will return a list of elements (each news are in a div.story tag). Then, for each news, you indicate how to retrieve the title within this ''… div.story", eg: h2.esc-lead-article-title > a > span And finally, same for the content of the news: div.esc-lead-snippet-wrapper

Here is a full example of my google news brain file:

.. code:: yaml

---
  - name: "Google-news-en"
    signals:
      - order: "what are the latest news"
    neurons:
      - web_scraper:
          url: "https://news.google.com"
          main_selector: "div.top-stories-section div.section-content div.story"
          title_selector: "h2.esc-lead-article-title > a > span"
          description_selector: "div.esc-lead-snippet-wrapper"
          file_template: "templates/en_web_scraper.j2"

Then, as usual, you need to create the template to indicate to Kalliopé how you want to hear these info, here is an example:

.. code:: jinja

{% raw %}
{% if returncode != 200 %}
    Error while retrieving web page.
{% else %}
    {% for g in news: %}
        Title: {{ g['title'] }}
        Summary: {{ g['content'] }}
    {% endfor %}
{% endif %}
{% endraw %}

Neuron should work on any pages. Main drawback here is that you are dependent of the site HTML, which is often more changing than an RSS feed or an API… but at least that does the job :)

Kalliope neuron for google calendar

Introduction

Even though I'm not a big fan of using Google (or any GAFAM) products, I must use this one for managing some of my meetings.

So, I worked and published a neuron that allows Kalliope to talk to google calendar so that kalliope can tell me what are my next meetings. The idea of the neuron is very simple and only work on read only.

Installation

Configuring your google calendar

First, you need to create an app in your google developer console and allow the google calendar API. For this you can go directly via this wizzard

or follow the step 1 of this tutorial

Copy / pasting the relative steps here:

  • Use this wizard to create or select a project in the Google Developers Console and automatically turn on the API. Click Continue, then Go to credentials.
  • On the Add credentials to your project page, click the Cancel button.
  • At the top of the page, select the OAuth consent screen tab. Select an Email address, enter a Product name if not already set, and click the Save button.
  • Select the Credentials tab, click the Create credentials button and select OAuth client ID.
  • Select the application type Other, enter the name of your app, and click the Create button.
  • Click OK to dismiss the resulting dialog.
  • Click the file_download (Download JSON) button to the right of the client ID.
  • Move this file to your working directory and rename it client_secret.json.

Installing Kalliope neuron

As any other neurons, just do .. code:: bash

  kalliope install --git-url https://github.com:bacardi55/kalliope-google-calendar.git

And now, you need to configure the neuron to make it works. The way I'm doing it, is:

  • Create a hidden directory in my kalliope's home for the google calendar secret files (do not put them in a public git repo ^^).

.. code:: bash

mkdir /home/pi/.google_calendar
mv /home/pi/client_secret.json ~/.google_calendar/
  • Configure your brain: .. code:: yaml


    • name: "Google-agenda-next" signals:
      • order: "what are my next meetings" neurons:
      • google_calendar: credentials_file: "/home/pi/.google_calendar/credentials.json" client_secret_file: "/home/pi/.google_calendar/client_secret.json" scopes: "https://www.googleapis.com/auth/calendar.readonly" application_name: "App name" max_results: 3 locale: fr_FR.UTF-8 # needs to be an installed locale file_template: "templates/fr_google_calendar.j2"

Remember to update the path if they are not the same and the application name as well. Your local needs to be installed as well on your system. And write your own template :)

Here is my example:

.. code:: jinja

{% raw %}
{% if count > 0 %}
    Your next meetings are
    {% for event in events %}
        on {{ event['time']['weekday'] }}, {{ event['time']['day'] }}, {{ event['time']['month'] }}, à {{ event['time']['hour'] }} hour, {{ event['time']['minute'] }} :
        {{ event['summary'] }}
    {% endfor %}
{% else %}
    You don't have any meeting coming up
{% endif %}
{% endraw %}

Limitation

Neuron is read only so you can't create calendar events from this neuron, but I'm always happy to integrate pull requests :)

Available commands on kalliope neuron

And we keep going with the blog post series about Kalliope :)

Why

Let's introduce here a simple neuron I developed that let you know what you can ask.

I know it might sound dumb, but it can be handy when other people leaving with you don't know as much as you your kalliope setup :) (Or as well, to make Kalliope answer when friends ask why i use it, I can make kalliope answers with the list of available orders).

This is why I developed the small neuron list_available_orders.

Installation:

As per the previous neuron, kalliope let you install via the command line the neuron:

.. code:: bash

kalliope install --git-url https://github.com/bacardi55/kalliope-list-available-orders.git

No additional library are required for this neuron.

usage

brain

Create you brain file, eg:

.. code:: yaml

---
  - name: "All-orders"
    signals:
      - order: "what can I ask"
    neurons:
      - list_available_orders:
          query_replace_text: "and the argument"
          ignore_machine_name: 1
          file_template: "templates/en_all_available_orders.j2"

The query_replace_text option is the text you want instead of "{{ query }}" (or any arguments). I put this option because otherwise Kalliope says open/close bracket which is a bit weird :)

template

You can use 2 variables in the template, nb_orders for the number of available orders and orders, a list of order. More description about the order here.

This template will simply make Kalliope speak out loud the number of orders and then all of them If no order are found, a simple "order not found" sentence will be said by Kalliope.

.. code:: jinja

{% raw %}
{% if nb_orders > 0 %}
    You have {{ nb_orders }} available orders:
    {% for order in orders %}
        {{ order }}
    {% endfor %}
{% else %}
    No order found.
{% endif %}
{% endraw %}

Kalliope community modules and the Picamera neuron

Kalliope has recently introduced the "community module" system that will help user share neurons, tts and stt easily via a simple git url.

This blog introduce quickly how it works and update the installation process of the picamera simple neurons to this method

Introducing the community neurons system

Goal here is to provide a way for the user to share their neurons. I believe the team are (or will) work on a "market place" for neurons, but as of today, the list is only available on the github doc page here, so you'll have to create a pull request to add your custom neuron in this list.

The neurons needs to be on a dedicated git repository and contains a dna.yml and a install.yml files on top of the python code of your neuron

dna.yml file contains the description of the module, including its name, the version of kalliope supported and some tags.

install.yml file contains an ansible playbook that will which include the dependencies that might need to be installed when installing the neuron (could be via pip or apt-get). It uses ansible behind the scene to do so.

More information about creating your neurons here.

Example with picamera neuron

So following the new rules, I've created a dedicated repository for the picamera neuron that can be found here. As the doc says, you just need to do

.. code:: bash

kalliope install --git-url https://github.com/bacardi55/kalliope-picamera

And it should install the picamera python library as well as cloning your repo in your neuron resource directory.

The neuron resource directory is set in the settings.yml like this:

.. code:: yaml

resource_directory:
  neuron: "resources/neurons"
  stt: "resources/stt"
  tts: "resources/tts"

Then, as before, you can create a brain to use this neuron like this:

.. code:: yaml

---
  - name: "Take-pictures"
    signals:
      - order: "take pictures"
    neurons:
      - camera:
          number_of_picture: 3
          directory: "/home/pi/Desktop/PIctures/"
          timer: 1
      - say:
          message: "Picture taken"

And voilà!

Kalliope Remote web app

Introducing Kalliope remote

While I still have to finish introducing the neurons I made for Kalliope, I took a bit of time since last week end to work on a side project for Kalliope: Kalliope-remote

It is a simple Single Page Apllication that leverage Kalliope API to display the list of available neurons and launch them via the web page.

![kalliope remote]({{ site.url }}/assets/kalliopeember.png)

The idea behind this app is to be able to set it up on my raspberry pi touchscreen where Kalliope is installed so that user that don't know available orders can look for it and even fire them without talking to Kalliope. I will also be able to use it from my phone later on as well as it can be useful in case i'm not near the mic but the audio is loud enough :)

How

Additional notes

Edit 30th of December: The PR has been validated and merged into core. Only thing needed is to configure Kalliope to allow CORS request. Here is explained how.

At the moment, the REST API of Kalliope doesn't allow request from different origin, thus blocking the request from the JS app (even if both are the same server, as port are different). To fix this, I've commited this patch which add an option in the settings file to allow CORS requests.

I'm planing on adding another pull request later to add a configurable list of origin, so keep looking github ^^.

As for the integration of the PR into the main repo, it might be on standby while waiting for possible move to Django (see discussion in the pull request thread)

What it does now

Basically, when the app loads it will request the list of synapse from your Kalliope set up and will display them nicely on the page.

  • List all synapses available on your kalliope instance
  • Allow you to launch synapse that has no parameter in the order

What's next ?

  • Managing orders with parameters inside
  • Better test coverage ^^
  • Leveraging the browser local storage to allow ignoring synapses (so you list only the one you desire)
  • Managing other type of signals (event)

It's still early project but does already allow you some remote management :)

Kalliope - Raspberrypi camera neuron

EDIT: Installing "community" modules like neuron is now a feature of Kalliope (via the CLI), please read here instead

To start with the kalliope blog posts series, I'll start with introducing the few neurons I created on top of the existing ones.

Simplest neuron I created is a neuron to leverage the raspberrypi camera to take picture when asking kalliope to do so. Kalliope is installed on a raspberrypi v3 with a touchscreen display and a pi camera

The neuron can be found here (I need to move each neurons on it's own github repository as their will be a market place for kalliope neuron soon).

To enable this neuron, you need to install first the picamera python library apt-get install python-picamera should do it.

Then, create your neuron file picamera.yml like the example one:

---
  - name: "Take-pictures"
    signals:
      - order: "take pictures"
    neurons:
      - camera:
          number_of_picture: 3
          directory: "/home/pi/Desktop/PIctures/"
          timer: 1
      - say:
          message: "Picture taken"

You can configure how many pictures Kalliope will take (3 in this example), where to save then (here /home/pi/Desktop/Pictures) and how many seconds between the picture Kalliope needs to wait.

As you can see, pretty simple, but for me it was more a poc of using the picamera with Kalliope, I have further idea for it in the future, maybe using OpenCV with it :).

But I have other neurons do upgrade / create before going this road.

Stay tune for the other neurons.