At Askia we love to talk about Askia things… and about a year ago, the technical team got together in a room and agreed on what was our biggest need: the ability to elegantly call a web service from a survey and decipher the result and store it appropriately.

Web-service not included

I have mentioned in previous articles how an API allows you extend your para-data. With the IP-address that you collect (and that we encrypt – GDPR is watching you), you can obtain the general location of the person. With the location, you can get the weather at the time of the interview and the likelihood they voted to a given party in the last elections.

You could always call a web service by adding some JavaScript in your page but that was not very elegant… and also made it hard to hide any authentication method.

So we decided to create a new routing where the Web Service was called from the server and not from the browser – effectively hiding the call from the interviewee. We got inspiration from the Postman interface and quickly put together a new routing.

The interface allows you to run different scripts depending on the success of the call and to manipulate and store the different parts of the response… and we introduced a new keyword CurrentHttpResponse.

At that point, we thought that this had been relatively easy and we contemplated a well deserved visit to the local pub for refreshments.

XML and the Argonauts

As we were putting together an example – calling openweathermap.org to get the weather anywhere in the world – we hit our first problem.

The response looked like like this:

<?xml version="1.0" encoding="utf-8"?>
<current>
   <city id="6690581" name="Belsize Park">
      <coord lon="-0.18" lat="51.55"></coord>
      <country>GB</country>
      <sun rise="2018-03-06T06:33:58" set="2018-03-06T17:50:36"></sun>
   </city>
   <temperature value="282.33" min="281.15" max="283.15" unit="kelvin"></temperature>
   <humidity value="66" unit="%"></humidity>
   <pressure value="988" unit="hPa"></pressure>
   <wind>
      <speed value="2.1" name="Light breeze"></speed>
      <gusts></gusts>
      <direction value="200" code="SSW" name="South-southwest"></direction>
   </wind>
   <clouds value="40" name="scattered clouds"></clouds>
   <visibility value="10000"></visibility>
   <precipitation mode="no"></precipitation>
   <weather number="521" value="shower rain" icon="09d"></weather>
   <lastupdate value="2018-03-06T13:50:00"></lastupdate>
 </current>

To get the temperature, we would have had to look for the string “temperature value=” and extract the following digits… it was possible but a bit of a dirty hack, we felt. As stated before, at Askia we love to talk but we hate dirty hacks.

So we started talking about having a XML parser. The cool kids in the dev team took a clear stand: we do not need a XML parser and we would be a laughing stock if we implemented one. What we needed was a JSON parser. Even better we thought: what if AskiaScript could natively support JSON? Note: I can confirm it, we did a XML parser anyway – I hope you are not laughing.

JSON native and the dictionary

So we came up with the following syntax:

Dim myAuthorVar = @{
 “name”:”Jerome”,
 “age“:21,
 “occupation”:”laughing stock”,
 “busy”: true,
 “children”: [“Mackenzie”, “Austin”],
 “address” : {
    “postcode”:”SW12”
    “city”:”london”
    }
 }

Return myAuthorVar [“occupation”]

We were very excited but that meant we need a new variable type – it’s sometimes called an object or a map but also a Dictionary – the failed librarians and encyclopaedists that we are loved that… so there it was: the Dictionary. It allows to store a series of named values in one object. You can set its properties with a method Set like this myAuthorVar. Set (”Busy”, False ). And access them like you would with an array but by specifying a string instead of a number like this: myAuthorVar [“name”].

Variant and Arrays of Variant

I mentioned that it would be a good time to go to the pub when somebody asked what was the type returned by a dictionary accessor. In other words what was the type of myAuthorVar [“age”] ? The response to this is “it depends”… and there was no way of knowing before. Right now, it was a number, but if a web service had indicated “age” as “fifty-ish”, the result would be a string.

So we had to introduce a new type: the Variant

If you called myAuthorVar.TypeOf(), it would return “variant”… but inside the variant is a dictionary. So we created a method for Variant to know what was inside and we called it InnerTypeOf. myAuthorVar.InnerTypeOf() does return “dictionary”.

It was also nice to write @[1 ,2 ,3] or even @[3.14159,”pear”, “apple”]  – both are arrays of variants that we decided to call “arrays” for simplicity.

A variant could hold any of what we decided to call the base basic types: number, string, date, dictionary and any array of the types above. OK – let’s go to the pub! But then we remembered that JSON supported Null and Booleans… and because we wanted full compatibility, we had to create two new AskiaScript types: Null (which does not do much) and Boolean having the possibility of only taking two values: true or false.

Booleans and back compatibility

This was a can of worms – because we used to consider True and False to be numbers. Let’s imagine some script like this:

 Dim myVariable = (Q1_Name = 7)
 ' … some clever coding…
 myVariable = 42
 ' … more clever coding…
 If myVariable = 42 Then
    ' Save the world ...
 Endif

In classic AskiaScript, this would create a variable called myVariable as a number with a value of 1 or 0 and later taking the value 42 allowing the world to be saved.

We did not want to break back compatibility. I am going to summarize what was hours of discussions. We decided that comparators (like equal or Has) had to return numbers. If they returned booleans, the setting to 42 would now trigger an error because 42 is not a Boolean. And if we permitted an automatic conversion of numbers into Booleans, my Variable would take the value True (and not 42) which would change the way the scripts ran… and the world as we know would perish.

Wordy woes

Having spoken for so long, we were quite thirsty as you might guess. But we realised that our language would become very verbose and somehow inelegant if we had to convert Variants into the type we wanted whenever we wanted to use them.

In the example above, if we wanted to find out the length of our author’s post-code, we would have had to write:

 Dim hisAddress = myAuthorVar["address"]
 Dim hisAddressAsDic = hisAddress.ToDictionary()
 Dim hisPostcode = hisAddressAsDic ["postcode"]
 Dim hisPostcodeStr = hisPostcode.ToString()
 return hisPostcodeStr.Length

This was ridiculous… it would take ages to write any serious code… and we had better things to do than write verbose code (at that stage I was thinking of all the beers I would not be able to drink if I had to type that much to get my own postcode). So we went back to the drawing table and agreed that

myAuthorVar [“address”] [“postcode”]. Length was all we needed.

This elegant code was only possible if Variants supported ALL the properties and methods of ALL the basic types. That was a lot of unit tests that we had to do. So we focused (no blurred vision) and we wrote them.

This meant a serious rewrite and a careful management of conflicts: Format is a method for numbers and dates and they act very differently. So we put together a set of rules.

I’ll give you a reference

At that point, we had spent a lot of time on this, we were (very) thirsty but we wanted it to be perfect. And we realised we had a problem – what if we wanted to change the Postcode of our author (by code).

myAuthorVar [“address”] returned a Variant holding a dictionary with the address – a copy of the address. So to change the postcode we would have needed to write:

 Dim hisAddress = myAuthorVar["address"]
 hisAddress.Set ( "postcode" , "EC2A" )
 myAuthorVar.Set ("address", hisAddress)

This was again way too verbose. So we decided that accessors ( the closed brackets [ ] used by dictionary and arrays) would not return a copy of the address but a reference to the address of the author. This meant that we could write

 myAuthorVar["address"].Set ( "postcode" , "EC2A" )

This added very serious complication the the code  it’s called pointers as in dangling pointers in C++)… and that’s very difficult to make work. In the above example (as in life), the variable hisAddress can outlive myAuthorVar. We had to write a lot of unit tests to ensure that everything worked and we did not have memory leaks. This is discussed here.

For short, a variable stops being a reference as soon as you assign it something else.

AskiaScript Anonymous

We had an ongoing problem with the Value property of question – and we thought it’d be a good idea to address it now before we went to the pub.

Q1.Value returns a string if Q1 is an open-ended question. And it returns an array of numbers if Q1 is closed with multiple response. It can also be a number or a date…

Now let’s imagine we have a script like this

 Dim myVariable = Q1
 ‘ On Mondays at precisely 12 o’clock
 If Now. Day()= 1 and Now.Hour() = 12 Then
    myVariable = Q2
 Endif
 ‘ What is myVariable.Value here?

AskiaScript is a compiler – it wants to know the type of things before it’s run… but in that example, myVariable. Value could be of a different type depending on the day and time it was run.

And what if we had something like Q1.NextVisibleQuestion.Value?

So we decided that as soon as you put a question into a variable then the variable becomes an “anonymous question”. All methods of an anonymous question would work but the Value property would be a variant…. And we also decided to make sure that CurrentQuestion was an anonymous question. Problem solved! Drinks anyone?

But then we had another huge back-compatibility problem. Let’s look at the following code:

 
 Dim myVariable = QNumeric
 Return myVariable + 1

In classic AskiaScript, the system would add an invisible “.Value” after QNumeric (we call that an implicit property). myVariable would be a number and we would return that number incremented by 1.

But with the introduction of anonymous questions, myVariable was now a question. Facing an operator (the +) we would add again the implicit property .Value. But now value would be a variant and we had no rule to add a variant to something else… up to now.

So we made sure that we had rules to add any variant to another Variant – or any basic type or array of basic types. Not just add but also subtract, multiply, divide, compare – including all the keywords like  Has, HasNone etc. In total, combining 4 operators, a dozen of comparators, with 6 basic types and 3 types of arrays (number, string and variants) that made a lot of decisions to take (and a lot of discussions) and many many unit tests.

Before we started this development, we had 1667 unit tests ensuring that all functions of the AskiaScript behave the same from one version to another.

For this, we had to add 2231 (!) more unit tests. Once they all passed successfully, we added the whole thing to Suite 5.4.8 and we hope you’ll like it.

Enough Quant Tricks, we’ll be in the pub for a swift one – we deserved it.