Big Data with just one digit

I know some of you think I only attend conferences for the free food, the drinks and the social scene. They are right – no point in me denying. But in-between parties, I tend to heal my hang-overs in the semi-darkness of conferences.

Coming back from the ASC and ESOMAR, there are a few new tendencies in the Autumn/Winter MRX fashion. Forgotten MROC, gamification, mobile research, Big Data – that’s so last year… it’s main stream, dude.

These days the cool kids talk about Automation, Data fusion, Artificial Intelligence… and the Tinderisation of research.

Automation – if you’re an assiduous reader of this blog, you know it’s coming and fully available at a software provider near you. I am not going to ramble anymore about this for now but watch this space.

Artificial Intelligence is the next big thing in Research. It has been successfully used to post-code and (less successfully) to measure sentiment in open-ended questions and tweets. It’s also good at recognising logos and objects on pictures and films, building accurate predictive models and beating me at Go (well the latter is not news and not strictly research)… but now AI is also used to merge data. There is an inconvenient truth about convenience panels… and MR data in general. If your survey is 40 minutes long (or 20 minute on a mobile device), the resulting data will be awful: the participants are either too unusual to be trusted or they don’t care because they are not incentivised.

Although there is no evidence of the length of surveys diminishing (according to SSI), every-one agrees that it needs to happen. One way is to… well… make up data. You do not ask all the questions to every-one and you copy the data around for similar looking interviews – this is called ascription (it has been around for some time). For you stat geeks out there, it’s traditionally done using the Mahalanobis distance. The new thing is to use machine learning to infer missing data. Mike Murray and James Eldridge from Research Now had a great paper about automating the splitting of surveys in chunks from their XML definition. Annelies Verhaeghe from Insites and John Colias from Decision Analyst also presented two great papers about enriching surveys with open big data.

And finally after the uberisation of research which has seen the arrival of monkeys, gizmos, nuts & limes, the new trend is the tinderisation of research. Millenials (there were boos whenever the term was used – and that was every 47 seconds) take decisions with their index. Left means no, right means yes… and survey research should follow. It’s easy to understand, fast to answer and it’s your system 1 talking… And the index is not just for decisions… the navigation of a survey should be done through flicks of the index. Almost being a millennial myself (the NSA has the names of those who are laughing), I see the attraction… and we are soon to release something code-named Jupiter that might just turn (or keep) Askia the best software for the Generations Y and Z.

Playing with D3js and AskiaVista

As you may already know, we added support for the Highcharts visualisation library as of version 6.0 of askiavista; and while it provides all the necessary visualisation needs for our online data analysis application, we had designed askiavista’s API to be agnostic when it comes to charting libraries.

From our initial use of NevronDundas Charts in version 5 to our use of Google Charts in one our initial demos of askiavista’s AJAX capabilities, we have always wanted to provide our users with the flexibility to chose their preferred charting library.

One such library we have always wanted to play around with is the amazing open source D3js visualisation components. This JavaScript library that produces dynamic, interactive data visualisations in web browsers. It makes use of the widely implemented SVG, HTML5 and CSS standards. While, D3 provides all basic types of charting components (bar, column, pie, area, …), it also provides many more advanced visualisations.

We have therefore played with both of those types of charts; from simple visualisations:

Classic bar chart

Diverging Stacked Bar Chart

… to more complex and advanced charts:

Sunburst Chart

Bubble Cross Tab

These Lab type demos are hooked up from a classic Askia survey via askiavista’s API to the D3 components. While D3 is not integrated by default in our API tooling, it is a simple proof-of-concept of such an implementation.

To view all the components we played around with, check out the full demo! Props to Brice de la Brière for making this possible!