How is ACTNext using AI and ML to transform the learning, measurement, and navigation landscape?

In this episode of the Navigator podcast, we discuss some of the artificial intelligence (AI) timeline, past, present and future, with Saad Khan , director of AI and machine learning for ACTNext. We discuss “black box” perceptions of AI research and we’ll learn a little about how the AI/ML team is using deep learning to create new tools for measurement and learning.



The views and opinions expressed in this podcast are those of the authors only and do not necessarily reflect the official policy or position of ACT, Inc.

Podcast transcript:

[Saad Khan] We’re working on solving problems for which there isn’t an existing solution, so really pushing the frontier over here. What my team and I are doing over here. ACT is really the cutting edge of what’s being done in the field at large when it comes to the use of and in education.

[Adam Burke] That’s Saad Khan, the director of artificial intelligence and machine learning for ACTNext. You’re listening to the ACTNext navigator podcast. I’m Adam Burke.

Today’s episode is brought to you by Sphinx! Are you looking for automated passage generation?
Sphinx encodes text data to automate summarization, sentence recommendation, paraphrasing and more.
Sphinx, the natural language processor for creating English passages quickly.
Find out more about Sphinx at

We’re talking about AI and ML today with Saad Kahn. What is AI? Where is it going? Where did it come from? Welcome to the show. Saad, can you tell us a little about yourself?

[SK] I’m Saad Khan and I lead the AI and machine learning team within ACTNext. What myself and my team do is build out a new generation of learning and measurement systems that help us address our educational needs. We’re talking about building tools that can help us measure complex competencies like communication ability, collaborative problem-solving abilities for which is hard to come up with a traditional form of measure and or assessment. For instance of paper and pencil tests or even self assessments for instance rate yourself on a scale of one to ten. Are you a good team player or what have you those kinds of measures tend to have subjective biases at times and don’t necessarily scale up or transfer well from one context or the other.
What I believe is a better way to approach that problem is to have people demonstrate those complex skills and abilities in an ecologically valid fashion as in the real world and have computer systems be able to actually extract from those performances valid evidence that could be used to draw inferences about their abilities or skill gaps. That’s where the AI / machine learning comes into play and we can certainly have human experts perform that role but once again you know we have the issue of potentially subjective biases or the fact that it’s going to be hard to scale that up. But computer systems can replicate the processes, particularly AI machine learning based systems, can replicate those human expert processes to be able to really provide a transparent and repeatable way to be able to draw the same kind of inferences that human experts would on complex scales.
We also work on addressing some challenges like how do we automate or semi automate tasks like creating new educational content which very much is to this point being a manual, labor intensive process. So how do we use Sur to the art and national language understanding to be able to create new items? I could go into new assessments or create new content that could be used for comprehension analysis now perhaps you will create multimodal content and by that I mean not just content which is in the form of text but it could be video it could be audio and so forth. We have a rather large charter or purview in which we operate. The team has been making some really exciting progress on those multiple fronts over the last couple of years.

[AB] AI/ML deep learning, how do you explain the differences or how would you explain it to someone who doesn’t know AI and ml and some of the different fields?
[SK] Thank you for asking that question because many times there are misunderstandings outside the communities of researchers and developers who are deeply involved in the state of the art and the progress be made in AI. In fact, the way I would describe them is that machine learning and deep learning themselves are facets or subfields within AI.
AI being more of the umbrella term for a variety of different ways in which we’re trying to make machines demonstrate intelligent behavior or the what would seem to us as solving tasks that would require human-like intelligence. Generally speaking the more accepted description of AI is having computer systems that can perceive their environments to distill from that perception actionable knowledge that can be stored away and then applied in a variety of new different contexts to achieve a particular goal that might be set for that computer system or computer agent.
The bigger goal in AI, which is sometimes called general AI, is to replicate human-like intelligence in a variety of different complex problems. AI, which began way back in in the 50s, have had multiple cycles of boom and bust if you will and many examples of successes having commercial and industrial applications and going through multiple rounds of development phases focus on different aspects of AI or machine learning is a particular kind of artificial intelligence which is heavily data-driven and the idea with machine learning is to be able to look for and identify patterns in data that can actually be used to create help us create models which are predictive in nature. So that in the future if you receive data which is different from the data that was used to learn the pattern in the first place can we use those models to be able to make predictive decision as to what that data actually means.

[AB] It’s hard to describe what algorithms are doing as they’re kind of changing themselves. Can you talk about the black box?
[SK] From a very technical perspective I think there is something. This is a legitimate issue when we have machine learning and particularly the new generation of machine learning which is deep learning we’ll be talking about building incredibly complex mathematical models that can have on the order of hundreds of millions of various different parameters that taken input data and are able to perform many different layers of transformations on that data to be able to come up with a final output which has very high degree of accuracy.
But it’s very unsatisfying when we’re not able to really explain what is going on in those millions of computations that led to that final result – that is a very human question to ask. Because when we think of ourselves as making complex decisions when we faced with multiple choices we’re able to articulate our thought processes and how we get to the final decision which is very hard to replicate in machines right no. So I think it’s a it’s a legitimate question to ask about the black box systems but it would be unfair to say that that is an inherent shortcoming or limitation of AI systems. On the contrary AI scientists take that very seriously as a problem.
Over the last five years I would say there has been tremendous progress. We started about 10 years ago when the deep learning were excited to catch on but now we were able to actually disambiguate what different layers of those deep neural network what sort of knowledge representations they are developing and learning so progress is being made to make those black boxes a bit more transparent and I will also add to that that with machine learning systems with computer systems in general it is actually at least possible to open up that black box and trying and try to diagnose what’s taking place over there and draw inferences and interpretations from it which is very hard it takes a while it takes a long time I’m sure know what I was going to say is that that is very hard if not impossible to do with human beings okay one has to rely on trust yeah when we when we try to ascertain from other people’s behavior what might be going on in the mind it’s impossible to actually hope decipher that so in a way AI systems are a bit more transparent or at least amenable to transparent than human beings are and so I just wanted to point that out that while it’s very fair to ask about the issue of the comparisons being sometimes being viewed as black boxes there is actually from a scientific view the opportunity to be able to really make them transparent and clarify exactly what the decisions or the processes were towards a particular outcome.

[AB] You talked about that kind of the history that AI has gone through different phases of growth and recession and it made me think that they probably came up with a computer that could play tic-tac-toe and beat a human and then it became playing a slightly more complex game, chess or Go or the next thing is maybe self-driving cars. Does the definition kind of change? Has it changed over time? What impresses us as AI is now passe or is that fair?
[SK] Absolutely, in fact there’s a cynical saying that AI is whatever that hasn’t already been done. So every time there is a significant problem that AI is able to solve we almost sort of moved the goalposts and say obviously this is there is a clear solution to this surely this cannot be so back when you know the first AI systems using knowledge representation and predicate logic or used to solve say checkers or when we had IBM built deep blue to beat Garry Kasparov, the grandmaster of chess, those were at the time perceived as the holy grails of artificial intelligence and as soon as they were solved everybody basically said that this surely cannot be artificial intelligence.
When we think about solving problems like how do we use GPS systems to help cars navigate from one place to the other and there’s a popular algorithm called a star search to be able to solve that point-to-point navigation problem and it’s basically ubiquitous now and so the Holy Grails of AI have moved on.
So there you are absolutely spot on that our expectations of AI continue to evolve but what hasn’t really changed is the notion that AI really ought to be when it comes to in replicating human intelligence behaviors the need to be able to generalize from one context of the other – that still remains the general AI vision and call remains. I think and that’s something that we still some distance from and it’s really exciting to see the progress being made in the field at the rate next be speed as it is.
But I think most experts in the field would agree that we are certainly some significant ways off from getting to that general AI goal.

[AB] So we haven’t reached artificial intelligence maybe because this definition keeps changing and it sounds like you were saying artificial intelligence would truly be to take different contexts and be able to adapt from different situation to situation.
[SK] That’s right and there’s a technical term that sometimes is used in AI called supervised learning versus unsupervised learning that sort of gets to the heart of this.
Supervised learning, particularly in the machine learning context, means that the problem of identifying patterns in data and to be able to say that what this particular pattern really means, we need human intervention actually and that’s where the supervised comes in. A human expert would say here’s a lot of data and I’m going to label some subset of the data over here.
Take the example, you’ve got a collection of different colored walls. So here are 10 balls and 5 of them are green, 3 of them are yellow and the rest of blue. That’s human level annotation or labeling. Right now we can take data about that collection of balls to say, all right let’s take a picture of that or what have you, the data that you getting from this part of the picture over here then, it becomes actually a pretty straightforward machine learning problem to say this kind of data seems is. It’s got this particular interpretation to it and when I see data which has similar characteristics or signatures as what is called green balls can be labeled as a green ball, that is the paradigm in which most of machine learning actually works. It’s called supervised training okay but as you can anticipate over here you know the issue the bottleneck is that we need that human intervention to be able to provide that labeling a high-level interpretation of the data reverse-flash [AB] You’re guiding.
[SK] Exactly mr. human is over here helping the machine learn but the machine has not learned on its own that there is such a thing as a green ball or there is some representation of that data that could be interpreted as such the task.
The field that is focused on solving that problem is called unsupervised learning and that gets to the heart of being able to replicate that very human ability or that very intelligence marker to learn peculiar patterns of the data that ascribe to it a label or a knowledge representation that can then be applied in a different context and there’s this whole process itself of the learning without having much supervision is what has been eluding artificial intelligence as a field and the best guess is that being able to solve that problem when will be the key that unlocks our achievement of general intelligence.
[AB] How far away are we?
[SK] It’s very hard to say. The progress is being made very quickly but you know it’s very hard to say that we will have general level intelligence in the next five years or in the next ten years because it seems like one of those things for which they will be a breakthrough that happens tomorrow, next year, maybe never we don’t know.

[AB] I want to know about Saad Khan. Tell me about your background. What inspired you to get into computer science in the first place when you were young, growing up?
[SK] I was always science junkie if you will, a big fan of Carl Sagan, watching his Cosmos series, the only TV that I really loved, except for anime, was science documentary. I always was attracted to science, cosmology, biology, or even technology and computer science.
One of my favorite authors in sci-fi Arthur C. Clarke and when I first saw Space Odyssey 2001, Stanley Kubrick’s movie, it just totally blew me away. Not just the fact that it was just so scientifically plausible and accurately done and all of that the computer system HAL, how it demonstrated not just intelligence but many very human behaviors of jealousy and sometimes deception and so forth and to be able to ascribe to a computer system those kinds of abilities were fascinating to me.
But that didn’t mean that I just wanted to just jump in to computer science at the time but it was one of the things that really fascinated me.
It wasn’t until I got to college that I really started taking a computer science seriously. My senior thesis for my undergrad which by the way was at a university in Pakistan where I actually grew up, Ghulam Ishaq Khan Institute [of Engineering Sciences and Technology].
My senior thesis was on building an AI system that replicated ant-like colony intelligence in artificial robots or ‘bots, which we simulated in a computer system and so we essentially built back in the day, 2003, an artificial neural network that was trained to replicate ant-like behavior in a colony of bots and by demonstrating intelligent being like foraging for food or flocking behavior and that was really successful that really inspired me to pursue a PhD in computer science.
I applied for an RPC program in the U.S. in computer vision and I got in and started working at the University of Central Florida with one of the most influential scholars in the field, Mubarak Shah.
A lot of my work was around how do we take visual data captured in multiple sensors like mobile cameras and solve the problem of tracking a human behavior in real time and that’s like a particular problem it can be division and that I’ve worked on publishing that that’s essentially how I really got going and computer science and in machine learning.
Incidentally that happened to be the time where machine learning be cities really started to get into that and exploding into a field that made huge impacts in a is in general and I guess the timing was fantastic Here I am…

[AB] And you did a lot of identification or facial recognition.
[SK] Not so much facial recognition but analyzing human facial expressions. So one subfield in computer vision is to be able to develop algorithms that can understand human emotion as human beings do, by having face-to-face conversations. So you can tell when somebody’s smiling or scowling or looking unhappy or frustrated what happy and obviously machines are very dumb at that at some point not so long ago.
My work was focused on actually being able to build machine learning based models that can take in visual data videos essentially of people’s faces and say alright what sort of emotion are they expressing.

[AB] You got your PhD in 2008 and I want to know what’s changed since?
[SK] Many changes in computer science and machine learning or AI in general you know some of the common things that people refer to are things like the fact that we have cloud computing resources an explosion of data to use and all of that is true.
To me the biggest change has been the replicability of results which is a very same scientific thing.
When I was working on my PhD where the notion that you can build models which would work outside of just your own specific data sets and your own lab environments and shared the data sets as well as code for the public to be able to replicate your results was unfortunately women novel but that has changed tremendously in the last 10 years. It’s almost expected now that you can quickly share results, datasets, models, in an open source fashion and they go viral. That in many ways how impact of the work is measured and that has significantly accelerated the pace in which I think that the field has made progress. Where one group develops an incremental improvement on the state of the art and a particular kind of an algorithm they share it to the community a different group is able to replicate that result make further an enhancement to it rather than starting from scratch as they would have to otherwise this field as a whole benefits so that is a very I think it is a great thing for science and it’s a great thing for the field.

[AB] What’s the best part of your job or what brings me the most satisfaction?
[SK] I guess it really has to be that we’re working on solving problems for which there isn’t an existing solution so we’re really pushing the frontier over here. I don’t want to sound too boastful but I generally believe that what my team and I are doing over here at ACT is really the cutting edge of what’s being done in the field at large when it comes to the use of and in education some of the problems that we trying to solve over here for instance using multimodal analytics the combination of machine learning, natural language processing, computer vision to measure and remediate collaborate problem solving in online multiplayer games. There’s nothing like that that I know of that’s the community-at-large of the field at large is engaged in or using natural language generation tools like Sphinx to create new educational content.
I think it’s super exciting to be part of a team and a set of initiatives that are really groundbreaking.
In addition, one of the strengths of ACTNext is the multidisciplinary nature of the teams within ACTNext and ACT at large and the close collaborations that take place between those teams.
I have the advantage of brainstorming ideas with the team members who have expertise in psychometrics and learning Sciences and so forth and not just rely on say my own background in machine learning or for instance the expertise of team members within the AIML team on coming up with solutions that actually many times require a multidisciplinary approach and to add to that – this kind of high risk, sometimes I fail mindset and approach really can only be incubated when there is an institutional support and appetite for that kind of risk-taking and out-of-the-box thinking. Having leaders like Alina von Davier, who champions ACTNext and its approach to problem solving, and even our CEO Marten Roorda who really has been a major supporter of what ACTNext has been bringing to ACT has been a great boon for us and helping us create really the safety net to fall back on when we actually end up failing sometimes.

[AB] Well thank you Saad, thanks for coming.
[SK] You’re welcome.

[AB] That’s our show. If you’d like to find out more about some of the projects Saad’s AIML team is working on, like Sphinx, CRASE plus, or the CPS X project, please visit We’ll sign off today with a clip from 2001: A Space Odyssey soundtrack. This is the “Blue Danube.”