AI takes JEE (advanced) test, does well... but not enough for IIT seat

An artificial intelligence (AI) module, based on the model underpinning ChatGPT, scored well enough to be in the 80-90 percentile of India’s one of the toughest engineering college admission tests but did not fare well enough to clinch scores that would secure a seat in the premier Indian Institute of Technology (IIT) colleges, an experiment by IIT-Delhi researchers has claimed.

- Ashna Butani

The findings come at a time when AI-based tools are showing promise in cracking tests, such as the American graduate eligibility test SAT, and the quantitative Graduate Record Examination (GRE), raising questions about what it holds for the future of work, once powerful tools such as these become widely adopted.

The experiments with the joint entrance exam (JEE) advanced offer new insights about what such systems, called large language models (LLMs), can achieve, and struggle with.

“GPT-4 got around 35 percent of the questions right, which would put it somewhere in the top 80 to 90 percentile of students. It would have to be in the top 90-100 percentile in order to get into an IIT... It’s almost there,” said Daman Arora, one of the researchers, who was pursuing MTech at IIT-Delhi’s computer science department along with his colleague Himanshu Gaurav Singh when they collaborated on the project. Professor Mausam, a computer science professor at IIT-Delhi and the founding head of IIT’s Yardi School of artificial intelligence, oversaw this research.

To give the AI model the challenge, the duo created JEE Bench, with 515 pre-engineering mathematics, physics, and chemistry problems from the past eight editions of the IIT JEE-Advanced Exam.