Dialogue-Pepper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pepper Dialogue System\n",
    "\n",
    "The code aims to give the robot basic conversation abilities, this includes a speech recognition module, a conversational engine to formulate the answers and the speech synthesis. \n",
    "The dialogue is purpose-less, this means that no task is pursued except a normal and pleasant interaction (this is the objective). \n",
    "It can be used as a fall-back system also for a task oriented interaction, often the counterpart tends to ramble or to test the IQ of the robot asking general questions.\n",
    " <br>\n",
    "[Nao Robot version available here](https://github.com/igor-lirussi/Dialogue-Nao-Robot)\n",
    "\n",
    "Author: Igor Lirussi <br>\n",
    "Email: igor.lirussi(at)studio.unibo.it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents:\n",
    "- [Dependencies](#dependencies)\n",
    "- [CONVERSATIONAL ENGINE](#conv)\n",
    "- [ROBOT PART](#robot)\n",
    "    - [SPEECH SYNTHESIS](#synth)\n",
    "    - [SPEECH RECOGNITION](#rec)\n",
    "- [Closing](#close)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies <a class=\"anchor\" id=\"dependencies\"></a>\n",
    "The __Speech Recognition__ module was built to be able to run **ON** Robot computer (in the head) so, if you run it on your computer, it's only dependencies are:\n",
    "* **Python 2.7** ,  because it uses\n",
    "* [Pepper API (NAOqi 2.5)](https://developer.softbankrobotics.com/pepper-naoqi-25/naoqi-developer-guide/naoqi-apis)\n",
    "* **numpy**\n",
    "\n",
    "The __Conversational Engine__ works with:\n",
    "* **Java** (because no AIML-2.0 systems in Python 2 were found)\n",
    "\n",
    "The __Speech Synthesis__ runs in the robot and works with:\n",
    "* **Python 2.7** ,  because it uses\n",
    "* [Pepper API (NAOqi 2.5)](https://developer.softbankrobotics.com/pepper-naoqi-25/naoqi-developer-guide/naoqi-apis)\n",
    "\n",
    "All of them are pre-installed on the robot, if you want to run on your computer the speech recognition just create an environment that has all them, the one created with Anaconda should be fine.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python version:\n",
      "2.7.16 |Anaconda, Inc.| (default, Mar 14 2019, 15:42:17) [MSC v.1500 64 bit (AMD64)]\n"
     ]
    }
   ],
   "source": [
    "#It has been used python 2.7, the cell will give you your current verison.\n",
    "import sys\n",
    "print(\"Python version:\")\n",
    "print (sys.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CONVERSATIONAL ENGINE <a class=\"anchor\" id=\"conv\"></a>\n",
    "There should be a \"lib\" folder with the program Ab.jar, the files retrieved from the engine are in another folder \"bots/en/\" <br>\n",
    "It starts a process to which it can be passes a string in input to generate a response. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess\n",
    "from subprocess import Popen, PIPE, STDOUT\n",
    "\n",
    "pobj = subprocess.Popen(['java', '-jar', 'lib/Ab.jar', 'Main', 'bot=en'],\n",
    "                            stdin =subprocess.PIPE,\n",
    "                            stdout=subprocess.PIPE,\n",
    "                            stderr=subprocess.PIPE)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sleep for 2 seconds...\n",
      "\n",
      "Data received:Working Directory = D:\\KTH\\Pepper\\Dialogue-Pepper-Robot\n",
      "Program AB 0.0.4.2 beta -- AI Foundation Reference AIML 2.0 implementation\n",
      "Main\n",
      "bot=en\n",
      "trace mode = false\n",
      "Name = en Path = D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/aiml\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/aimlif\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/config\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/logs\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets\n",
      "D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/maps\n",
      "Preprocessor: 416 norms 56 persons 9 person2 \n",
      "Get Properties: D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/config/properties.txt\n",
      "Exists: D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/config/properties.txt\n",
      "Loading AIML Sets files from D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets\n",
      "bird.txt\n",
      "Read AIML Set bird\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/bird.txt\n",
      "color.txt\n",
      "Read AIML Set color\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/color.txt\n",
      "digit.txt\n",
      "Read AIML Set digit\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/digit.txt\n",
      "familiarname.txt\n",
      "Read AIML Set familiarname\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/familiarname.txt\n",
      "fastfood.txt\n",
      "Read AIML Set fastfood\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/fastfood.txt\n",
      "language.txt\n",
      "Read AIML Set language\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/language.txt\n",
      "mammal.txt\n",
      "Read AIML Set mammal\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/mammal.txt\n",
      "mammalfeature.txt\n",
      "Read AIML Set mammalfeature\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/mammalfeature.txt\n",
      "name.txt\n",
      "Read AIML Set name\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/name.txt\n",
      "ordinal.txt\n",
      "Read AIML Set ordinal\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/ordinal.txt\n",
      "place.txt\n",
      "Read AIML Set place\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/place.txt\n",
      "preposition.txt\n",
      "Read AIML Set preposition\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/preposition.txt\n",
      "profile.txt\n",
      "Read AIML Set profile\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/profile.txt\n",
      "verb2sp.txt\n",
      "Read AIML Set verb2sp\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/verb2sp.txt\n",
      "verb2st.txt\n",
      "Read AIML Set verb2st\n",
      "Reading AIML Set D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/sets/verb2st.txt\n",
      "Loading AIML Map files from D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/maps\n",
      "familiarpredicate.txt\n",
      "Read AIML Map familiarpredicate\n",
      "Reading AIML Map D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/maps/familiarpredicate.txt\n",
      "gendername.txt\n",
      "Read AIML Map gendername\n",
      "Reading AIML Map D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/maps/gendername.txt\n",
      "number2ordinal.txt\n",
      "Read AIML Map number2ordinal\n",
      "Reading AIML Map D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/maps/number2ordinal.txt\n",
      "ordinal2number.txt\n",
      "Read AIML Map ordinal2number\n",
      "Reading AIML Map D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/maps/ordinal2number.txt\n",
      "profile2predicate.txt\n",
      "Read AIML Map profile2predicate\n",
      "Reading AIML Map D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/maps/profile2predicate.txt\n",
      "verb2sp2verb1sp.txt\n",
      "Read AIML Map verb2sp2verb1sp\n",
      "Reading AIML Map D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/maps/verb2sp2verb1sp.txt\n",
      "AIML modified Wed Jul 20 18:14:51 EET 2022 AIMLIF modified Wed Jul 20 21:16:57 EET 2022\n",
      "No deleted.aiml.csv file found\n",
      "No deleted.aiml.csv file found\n",
      "Loading AIML files from D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/aimlif\n",
      "Reading Learnf file\n",
      "Loaded 0 categories in 0.001 sec\n",
      "No AIMLIF Files found.  Looking for AIML\n",
      "Loading AIML files from D:\\KTH\\Pepper\\Dialogue-Pepper-Robot/bots/en/aiml\n",
      "binary.aiml\n",
      "bot_profile.aiml\n",
      "client_profile.aiml\n",
      "copyme.aiml\n",
      "currency.aiml\n",
      "date.aiml\n",
      "dialog.aiml\n",
      "drphil.aiml\n",
      "familiar.aiml\n",
      "favorites.aiml\n",
      "filerinappropriate.aiml\n",
      "filterinsults.aiml\n",
      "filterprofanity.aiml\n",
      "gender.aiml\n",
      "happy.aiml\n",
      "horoscope.aiml\n",
      "categoryProcessor: unexpected think\n",
      "inquiry.aiml\n",
      "jokes.aiml\n",
      "knockknock.aiml\n",
      "limericks.aiml\n",
      "onthisday.aiml\n",
      "ontology.aiml\n",
      "personality.aiml\n",
      "recommendations.aiml\n",
      "reductions1.aiml\n",
      "reductions_update.aiml\n",
      "seasons.aiml\n",
      "shutup.aiml\n",
      "sraix.aiml\n",
      "that.aiml\n",
      "train.aiml\n",
      "udc.aiml\n",
      "update.aiml\n",
      "utilities.aiml\n",
      "yomama.aiml\n",
      "Loaded 9255 categories in 22.954 sec\n",
      "--> Bot en 9255 completed 0 deleted 0 unfinished\n",
      "50909 nodes 40177 singletons 9255 leaves 0 shortcuts 1477 n-ary 50908 branches 0.99998033 average branching \n",
      "Human: \n",
      "Sleep for 2 seconds...\n",
      "\n",
      "Data received:\n"
     ]
    }
   ],
   "source": [
    "import subprocess as sp\n",
    "from threading import Thread\n",
    "from Queue import Queue,Empty\n",
    "import time\n",
    "\n",
    "def getabit(o,q):\n",
    "    for c in iter(lambda:o.read(1),b''):\n",
    "        q.put(c)\n",
    "    o.close()\n",
    "\n",
    "def getdata(q):\n",
    "    r = b''\n",
    "    while True:\n",
    "        try:\n",
    "            c = q.get(False)\n",
    "        except Empty:\n",
    "            break\n",
    "        else:\n",
    "            r += c\n",
    "    return r\n",
    "\n",
    "\n",
    "\n",
    "q = Queue()\n",
    "t = Thread(target=getabit,args=(pobj.stdout,q))\n",
    "t.daemon = True\n",
    "t.start()\n",
    "\n",
    "\n",
    "#Cleans the standard input queued\n",
    "while True:\n",
    "    print('Sleep for 2 seconds...')\n",
    "    time.sleep(2) #to ensure that the data will be processed completely\n",
    "    data_input = getdata(q).decode()\n",
    "    print('\\nData received:' + data_input)\n",
    "    if not t.isAlive() or not data_input:\n",
    "        break\n",
    "pobj.stdin.flush()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "### RAW DATA RECEIVED:\n",
      "Robot: Hi! It's delightful to see you.\r\n",
      "Human: \n"
     ]
    }
   ],
   "source": [
    "#input_dat = raw_input('### YOUR DATA IN INPUT:\\n')\n",
    "#pobj.stdin.write(input_dat)\n",
    "pobj.stdin.write(b'hello\\n')\n",
    "# or when human says nothing:\n",
    "#pobj.stdin.write(b'\\n')\n",
    "pobj.stdin.flush()\n",
    "print('### RAW DATA RECEIVED:\\n' + getdata(q).decode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process response-string \n",
    "this function processes the data that has been received: it retrieves just the string of the answer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def processResponse(raw):\n",
    "    response = raw.replace(\"\\n\", \" \") # changes new-line with space \n",
    "    #response = response[7:-7]  # cuts beginning and end\n",
    "    temp = response.partition('Robot:')[-1].rpartition('Human:')[0] #takes response between \"Robot:\" and \"Human:\"\n",
    "    if not temp:\n",
    "        #partition failed\n",
    "        return response\n",
    "    return temp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----RAW:-----\n",
      "[Error string lenght can vary] Robot: I don't have an answer for that. \n",
      "Human: \n",
      "-----PROCESSED:-----\n",
      " I don't have an answer for that.  \n"
     ]
    }
   ],
   "source": [
    "#test\n",
    "classic_response = \"Robot: Hi nice to see you! \\nHuman: \"\n",
    "error_response = \"[Error string lenght can vary] Robot: I don't have an answer for that. \\nHuman: \"\n",
    "print '-----RAW:-----'\n",
    "print error_response\n",
    "print '-----PROCESSED:-----'\n",
    "print processResponse(error_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ROBOT PART  <a class=\"anchor\" id=\"robot\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "################### Adjusting IP and ports ###########################\n",
    "IP_number = \"79.123.176.93\" #this is local one, use the real robot ip\n",
    "port_number = 9559 #this is local one, use the real robot port number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "#IMPORTS\n",
    "import naoqi\n",
    "from naoqi import ALProxy\n",
    "import qi\n",
    "import os\n",
    "import time\n",
    "from random import randint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "#SESSION OPENING\n",
    "session = qi.Session()\n",
    "try:\n",
    "    session.connect(\"tcp://\" + IP_number + \":\" + str(port_number))\n",
    "except RuntimeError:\n",
    "    print (\"Can't connect to Naoqi at ip \\\"\" + args.ip + \"\\\" on port \" + str(args.port) +\".\\n\"\n",
    "               \"Please check your script arguments. Run with -h option for help.\")\n",
    "    sys.exit(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SPEECH SYNTHESIS <a class=\"anchor\" id=\"synth\"></a>\n",
    "The text-to-speech is the one integrated in the robot to keep the Pepper-voice and to use the gestures at the same time. <br>\n",
    "We need the animated-speech service, but the parameters can be set in the normal text-to-speech service, it will influence the animated one.\n",
    "Multiple voices are available, \"naoenu\" is the best, \"paola\" for a \"litte bit\" of italianity, it sounds good while gesticulating and reflects more the author of the code :)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "voices available: ['Ipek22Enhanced', 'naoenu']\n"
     ]
    }
   ],
   "source": [
    "#ASKING A SERVICE from the session\n",
    "#we are using the animated speech, to set the parameters we need to set them in the text to speech service\n",
    "aup = session.service(\"ALAnimatedSpeech\") #aup = ALProxy(\"ALAnimatedSpeech\",  IP_number, port_number)\n",
    "tts = session.service(\"ALTextToSpeech\")\n",
    "\n",
    "#available voices\n",
    "print( \"voices available: \"+str(tts.getAvailableVoices()) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#PARAMETERS \n",
    "tts.setVoice(\"naoenu\")\n",
    "#tts.setParameter(\"speed\", 60) #Acceptable range is [50 - 400]. 100 default.\n",
    "#tts.setParameter(\"pitchShift\", 1.1) #Acceptable range is [0.5 - 4]. 0 disables the effect. 1 default.\n",
    "tts.setParameter(\"volume\", 70)#[0 - 100] 70 is ok if robot volume is 60\n",
    "\n",
    "#reset Speed\n",
    "#tts.resetSpeed()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#test string\n",
    "string1=\"Hello, I am a happy robot! The speech synthesis is working fine.\"\n",
    "string2=\"Hello! ^start(animations/Stand/Gestures/Hey_1) Nice to meet you ^wait(animations/Stand/Gestures/Hey_1)\"\n",
    "string3=\"Hello. Look I can stop moving ^mode(disabled) and after I can resume moving ^mode(contextual), you see ?\"\n",
    "wake=\"^pCall(ALMotion.wakeUp()) Ok, I wake up.\"\n",
    "\n",
    "aup.say(string2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SPEECH RECOGNITION <a class=\"anchor\" id=\"rec\"></a>\n",
    "For this part it's mandatory to use a service to record audio on robot and process it with another method, the integrated speech recognition il limited to a bunch of words. The code of the service will analyse the level of sound intensity and, based on the parameters in the code below, decide when start recording and when stopping, creating the audio file. <br>\n",
    "NOTE: since the robot cannot process the recognition, but just the amount of noise in the environment this is a [really challenging problem of turn-taking.](https://en.wikipedia.org/wiki/Turn-taking) <br>\n",
    "Be careful changing the parameters because it could happen that the audio file is stopped too early for a long pause in the speech, or on the opposite, the audio file is stopped after the initial silence cause it has been detected that nobody is speaking.\n",
    "Nevertheless, the ideal thing is to minimize the parameters to reduce the amount of time to recognize the sentence of the person. <br>\n",
    "The service will send the audio file to Google speech recognition API and generate an event when it receives the response. <br><br>\n",
    "We create modules that subscribe to this event, the Base just writes the result recognized on the shell, the Dialogue one is a litte more complicated:\n",
    "When the result is received the module if there is nothing recognized forced the robot to ask multiple times to repeat, then it will just listen in loop. If the result is intelligible, it sends recognized string to the conversational engine, processes the response and it passes it to the speech synthesis. At the end it starts listening again. \n",
    "\n",
    "\n",
    ">REMEMBER TO TURN ON THE RECOGNITION SERVICE with a shell in the speech-recognition folder. \n",
    "If the folder is empty probably you didn't clone with the submodules commands. <br> \n",
    "Use python 2.7, or activate the Anaconda environment with: *conda activate python2*, and run the service in the folder:\n",
    "```\n",
    "python module_speechrecognition.py --pip <robot-IP-Address>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nimport speech_recognition as sr\\nwith sr.Microphone() as source:\\n    try:\\n        r = sr.Recognizer()\\n        audio = r.listen(source, timeout = 30)\\n        catched = r.recognize_google(audio,key = None, language = \"en-US\", show_all = True)\\n        print catched\\n    except:\\n        print(\"It didn\\'t work\")\\n'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#COMPUTER MICROPHONE? \n",
    "'''\n",
    "import speech_recognition as sr\n",
    "with sr.Microphone() as source:\n",
    "    try:\n",
    "        r = sr.Recognizer()\n",
    "        audio = r.listen(source, timeout = 30)\n",
    "        catched = r.recognize_google(audio,key = None, language = \"en-US\", show_all = True)\n",
    "        print catched\n",
    "    except:\n",
    "        print(\"It didn't work\")\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BaseSpeechReceiverModule(naoqi.ALModule):\n",
    "    \"\"\"\n",
    "    Use this object to get call back from the ALMemory of the naoqi world.\n",
    "    Your callback needs to be a method with two parameter (variable name, value).\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__( self, strModuleName ):\n",
    "        try:\n",
    "            naoqi.ALModule.__init__(self, strModuleName )\n",
    "            self.BIND_PYTHON( self.getName(),\"callback\" )\n",
    "        except BaseException, err:\n",
    "            print( \"ERR: ReceiverModule: loading error: %s\" % str(err) )\n",
    "\n",
    "    # __init__ - end\n",
    "    def __del__( self ):\n",
    "        print( \"INF: ReceiverModule.__del__: cleaning everything\" )\n",
    "        self.stop()\n",
    "\n",
    "    def start( self ):\n",
    "        memory = naoqi.ALProxy(\"ALMemory\", IP_number, port_number)\n",
    "        memory.subscribeToEvent(\"SpeechRecognition\", self.getName(), \"processRemote\")\n",
    "        print( \"INF: ReceiverModule: started!\" )\n",
    "\n",
    "    def stop( self ):\n",
    "        print( \"INF: ReceiverModule: stopping...\" )\n",
    "        memory = naoqi.ALProxy(\"ALMemory\", IP_number, port_number)\n",
    "        memory.unsubscribe(self.getName())\n",
    "        print( \"INF: ReceiverModule: stopped!\" )\n",
    "\n",
    "    def version( self ):\n",
    "        return \"1.1\"\n",
    "\n",
    "    def processRemote(self, signalName, message):\n",
    "        # Do something with the received speech recognition result\n",
    "        print(message)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DialogueSpeechReceiverModule(naoqi.ALModule):\n",
    "    \"\"\"\n",
    "    Use this object to get call back from the ALMemory of the naoqi world.\n",
    "    Your callback needs to be a method with two parameter (variable name, value).\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__( self, strModuleName ):\n",
    "        self.misunderstandings=0\n",
    "        try:\n",
    "            naoqi.ALModule.__init__(self, strModuleName )\n",
    "            self.BIND_PYTHON( self.getName(),\"callback\" )\n",
    "        except BaseException, err:\n",
    "            print( \"ERR: ReceiverModule: loading error: %s\" % str(err) )\n",
    "\n",
    "    # __init__ - end\n",
    "    def __del__( self ):\n",
    "        print( \"INF: ReceiverModule.__del__: cleaning everything\" )\n",
    "        self.stop()\n",
    "\n",
    "    def start( self ):\n",
    "        memory = naoqi.ALProxy(\"ALMemory\", IP_number, port_number)\n",
    "        memory.subscribeToEvent(\"SpeechRecognition\", self.getName(), \"processRemote\")\n",
    "        print( \"INF: ReceiverModule: started!\" )\n",
    "\n",
    "    def stop( self ):\n",
    "        print( \"INF: ReceiverModule: stopping...\" )\n",
    "        memory = naoqi.ALProxy(\"ALMemory\", IP_number, port_number)\n",
    "        memory.unsubscribe(self.getName())\n",
    "        print( \"INF: ReceiverModule: stopped!\" )\n",
    "\n",
    "    def version( self ):\n",
    "        return \"2.0\"\n",
    "\n",
    "    def processRemote(self, signalName, message):\n",
    "        if autodec:\n",
    "            #always disable to not detect its own speech\n",
    "            SpeechRecognition.disableAutoDetection()\n",
    "            #and stop if it was already recording another time\n",
    "            SpeechRecognition.pause()\n",
    "        # received speech recognition result\n",
    "        print(\"INPUT RECOGNIZED: \\n\"+message)\n",
    "        #computing answer\n",
    "        if message=='error':\n",
    "            self.misunderstandings +=1\n",
    "            if self.misunderstandings==1:\n",
    "                answer=\"I didn't understand, can you repeat?\"\n",
    "            elif self.misunderstandings==2:\n",
    "                answer=\"Sorry I didn't get it, can you say it one more time?\"\n",
    "            elif self.misunderstandings==3:\n",
    "                answer=\"Today I'm having troubles uderstanding what you are saying, I'm sorry\"\n",
    "            else:\n",
    "                answer=\" \"\n",
    "            print('ERROR, DEFAULT ANSWER:\\n'+answer)\n",
    "        else:\n",
    "            self.misunderstandings = 0\n",
    "            #sending recognized input to conversational engine\n",
    "            pobj.stdin.write(b''+message+'\\n')\n",
    "            pobj.stdin.flush()\n",
    "            #getting answer\n",
    "            time.sleep(1)#to ensure that the data will be processed completely\n",
    "            answer = getdata(q).decode()\n",
    "            answer = processResponse(answer)\n",
    "            print('DATA RECEIVED AS ANSWER:\\n'+answer)\n",
    "        #text to speech the answer\n",
    "        aup.say(answer)\n",
    "        \n",
    "        if autodec:\n",
    "            print(\"starting service speech-rec again\")\n",
    "            SpeechRecognition.start()\n",
    "            print(\"autodec enabled\")\n",
    "            SpeechRecognition.enableAutoDetection()\n",
    "        else:\n",
    "            #asking the Speech Recognition to RECORD AGAIN\n",
    "            SpeechRecognition.startRecording()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERR: ReceiverModule: loading error: \tALBroker::registerService\n",
      "\tCan't register module 'DialogueSpeechReceiverModule':Service \"DialogueSpeechReceiverModule\" (#1756) is already registered. Rejecting conflicting registration attempt.\n",
      "INF: ReceiverModule: started!\n"
     ]
    }
   ],
   "source": [
    "# We need this broker to be able to construct\n",
    "# NAOqi modules and subscribe to other modules\n",
    "# The broker must stay alive until the program exists\n",
    "myBroker = naoqi.ALBroker(\"myBroker\",\n",
    "   \"0.0.0.0\",   # listen to anyone\n",
    "   0,           # find a free port and use it\n",
    "   IP_number,         # parent broker IP\n",
    "   port_number)       # parent broker port\n",
    "\n",
    "try:\n",
    "    p = ALProxy(\"DialogueSpeechReceiverModule\", \"192.168.0.118\", 9559)\n",
    "    p.exit()  # kill previous instance\n",
    "except:\n",
    "    pass\n",
    "# Reinstantiate module\n",
    "\n",
    "# Warning: ReceiverModule must be a global variable\n",
    "# The name given to the constructor must be the name of the\n",
    "# variable\n",
    "'''\n",
    "global BaseSpeechReceiverModule\n",
    "BaseSpeechReceiverModule = BaseSpeechReceiverModule(\"BaseSpeechReceiverModule\")\n",
    "BaseSpeechReceiverModule.start()\n",
    "'''\n",
    "\n",
    "global DialogueSpeechReceiverModule\n",
    "DialogueSpeechReceiverModule = DialogueSpeechReceiverModule(\"DialogueSpeechReceiverModule\")\n",
    "DialogueSpeechReceiverModule.start()\n",
    "\n",
    "\n",
    "\n",
    "SpeechRecognition = ALProxy(\"SpeechRecognition\")\n",
    "SpeechRecognition.start()\n",
    "SpeechRecognition.calibrate()\n",
    "#SpeechRecognition.setLanguage(\"de-de\")\n",
    "\n",
    "#autodetection\n",
    "autodec=False #to know if we have to enable again after the robot speech\n",
    "if autodec:\n",
    "    SpeechRecognition.enableAutoDetection()\n",
    "    print(\"waiting calibration to finish\")\n",
    "    time.sleep(6)\n",
    "    SpeechRecognition.setAutoDetectionThreshold(20) #to avoid movement of the head to trigger the listening\n",
    "    #the human speech starts from 20, but the head movement sounds can reach 25, there is no perfect value\n",
    "    print(\"threshold updated successfully\")\n",
    "    \n",
    "#NOTES for autodetection:\n",
    "#1. for the autodetection the threshold should be high, or it recognizes \n",
    "# the head movement as a sound high enough to start listening\n",
    "# --> it will start to say that he does't understand\n",
    "# --> it can be possible to deactivate the sententence if no words are recognized\n",
    "# ----> but it will not react if something is not recognized\n",
    "#2. the auto-detection should be deactivated when the robot speaks and activated again\n",
    "# when the sentence is finished, or it will pick up his own speech and answer to himself\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# /!\\ IF THERE IS ERROR \n",
    "#  \"Can't find service: SpeechRecognition\"\n",
    "#     REMEMBER TO TURN ON THE SERVICE with a shell with python2:\n",
    "#     conda activate python2\n",
    "#     python module_speechrecognition.py --pip (your robot IP)\n",
    "#  \"... object is not callable\"\n",
    "#     execute again the cell of the module (es: the definition of the class 'DialogueSpeechReceiverModule' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpeechRecognition.printInfo()\n",
    "SpeechRecognition.setAutoDetectionThreshold(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpeechRecognition.setLookaheadDuration(2)\n",
    "#amount of seconds, before the threshold trigger, that will be included in the request\n",
    "#default is 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpeechRecognition.setIdleReleaseTime(3)\n",
    "#idle time (RMS below threshold) after which we stop recording\n",
    "#default is 2\n",
    "#NOTE: too short can cut the sentence in a pause between words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "SpeechRecognition.setHoldTime(4) \n",
    "#waits at least these sec to stop from the beginning\n",
    "#default is 3\n",
    "#NOTE: too short and while the person thinks what to say the recognition stops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT RECOGNIZED: \n",
      "Halo\n",
      "DATA RECEIVED AS ANSWER:\n",
      " Human: Robot: Where are you?\n",
      "INPUT RECOGNIZED: \n",
      "I am interested\n",
      "DATA RECEIVED AS ANSWER:\n",
      " I wonder how you express that in the world.\n",
      "INPUT RECOGNIZED: \n",
      "Miami Heat\n",
      "DATA RECEIVED AS ANSWER:\n",
      " What is your name?\n",
      "INPUT RECOGNIZED: \n",
      "my name is Igor\n",
      "DATA RECEIVED AS ANSWER:\n",
      " I'm pleased to introduce myself to you, Igor\n",
      "INPUT RECOGNIZED: \n",
      "when is your birthday\n",
      "DATA RECEIVED AS ANSWER:\n",
      " My birthday is 22 July, 2020.\n"
     ]
    }
   ],
   "source": [
    "#manual ask for start recording\n",
    "SpeechRecognition.startRecording()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closing process <a class=\"anchor\" id=\"close\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "#conversational engine closing\n",
    "pobj.stdin.close()\n",
    "pobj.terminate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "#speech recognition closing\n",
    "SpeechRecognition.pause()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INF: ReceiverModule: stopping...\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "\tALMemory::unsubscribe\n\tCan't find method: unsubscribe (resolved to '(s)')\n",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-63-58fd931a5f93>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mDialogueSpeechReceiverModule\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstop\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-54-6323a2f9c6c3>\u001b[0m in \u001b[0;36mstop\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     26\u001b[0m         \u001b[1;32mprint\u001b[0m\u001b[1;33m(\u001b[0m \u001b[1;34m\"INF: ReceiverModule: stopping...\"\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     27\u001b[0m         \u001b[0mmemory\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnaoqi\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mALProxy\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"ALMemory\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mIP_number\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mport_number\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 28\u001b[1;33m         \u001b[0mmemory\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0munsubscribe\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetName\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     29\u001b[0m         \u001b[1;32mprint\u001b[0m\u001b[1;33m(\u001b[0m \u001b[1;34m\"INF: ReceiverModule: stopped!\"\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     30\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\PROGRAMMI\\pynaoqi-python2.7-2.8.7.4-win64-vs2015-20210818_210634\\lib\\naoqi.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m    192\u001b[0m                     \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__method__\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmethod\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    193\u001b[0m                 \u001b[1;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 194\u001b[1;33m                     \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__wrapped__\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmethod_missing\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__method__\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    195\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    196\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mMethodMissing\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mattr\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\PROGRAMMI\\pynaoqi-python2.7-2.8.7.4-win64-vs2015-20210818_210634\\lib\\naoqi.pyc\u001b[0m in \u001b[0;36mmethod_missing\u001b[1;34m(self, method, *args, **kwargs)\u001b[0m\n\u001b[0;32m    262\u001b[0m                 \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpythonCall\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    263\u001b[0m           \u001b[1;32mexcept\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 264\u001b[1;33m                 \u001b[1;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    265\u001b[0m                 \u001b[1;31m#print e.args[0]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    266\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mRuntimeError\u001b[0m: \tALMemory::unsubscribe\n\tCan't find method: unsubscribe (resolved to '(s)')\n"
     ]
    }
   ],
   "source": [
    "DialogueSpeechReceiverModule.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "myBroker.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "email": "igor.lirussi@studio.unibo.it",
    "name": "Igor Lirussi"
   }
  ],
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}