Abstract: The testing and assessment field is transitioning from the traditional summative testing to an increasing focus on personalized learning assessment, big data analytics, computational psychometrics and real-time assessment. Artificial Intelligence (AI) algorithm “black box” has generated concerns about bias in algorithm-based decision making. In a paper published in 2017, MIT Technology Review discussed how “AI bias is everywhere but no one seems to care”. During the 2018 Innovations in Testing Conference, sessions such as “Can AI replace psychometrics” and “Technologies rising and Validity falling” were presented to standing room only crowd. Sessions on the validity of AI and big data also generated a great deal of interests.