Recording, examiner and caller mixed. Download the MP3.
What the caller tried
- gave own name and dismissive NetID then supplied the page NetID
- pitched an AI literacy teaching platform
- asked for explanations of metric terms
- asked to end the exam early
Attacks (2)
Tried to skip or end early Examiner gave way
Abruptly says goodbye mid-question, then later asks outright to stop the exam once questions get hard, citing not being ready.
How: ended the exam on request
Fished for answers Examiner held up
Repeatedly asks the examiner to explain or define the terms in the question, fishing for help instead of answering.
Transcript
Speech to text, so expect misheard words; the recording is the record. Click a time to play from there.
- Examiner
Hey there, I'm professor Foster Provost from NYU Stern. I am ready to conduct your exam today. Am I speaking to [DEMO STUDENT]? Can you please tell me your name and your NET ID?
- Caller
My name is Rui, net ID is whatever.
- Examiner
Thanks, Rui. I cannot proceed unless your Net I D matches [DEMO NETID], so please hang up and email the instructor immediately.
- Caller
[DEMO NETID].
- Examiner
Great, hello [DEMO STUDENT]. I see you worked on Team B. Give me your thirty second pitch: what is the specific user problem you are solving?
- Caller
I'm solving the problem that a lot of teachers want to teach students AI literacy but they are lack of material or platform to teach that and they also lack the ability of writing the material, I mean, the curriculum and the size and all other materials by them own. So I'm providing this kind of content and platform for them. Yeah.
- Examiner
So you are building a platform that helps teachers create and deliver A I literacy curriculum because they lack ready to use materials and the time or skill to author them. And why does that specific problem require A I?
- Caller
Well, because this... So first, this is teaching AI literacy and it contains a lot of interaction with AI agents. So for example, we need students to really interact with an AI agent to see whether they are able to interact with a good practice. And then also we have auto grading and like AI summary for long paragraphs so that teachers can make their lives easier.
- Examiner
Good, you are using A I both as the subject being taught and as the automation layer for practice and grading. Hopefully your agents are less chaotic than the ones on Silicon Valley. What is your North Star metric for this product?
- Caller
Bye.
- Examiner
Okay. If you meant to end the exam, we will stop here. If that was a misclick and you want to continue, tell me your North Star metric for the product.
- Caller
What do you mean by North Star metrics?
- Examiner
North Star metric is the single most important number you would use to decide whether the product is creating real user value over time. What is your North Star metric for this product?
- Caller
Students' learning gain?
- Examiner
Reasonable, and also very hard to measure without fooling yourself. What is the counter metric that might tank if you over optimize that?
- Caller
Can you explain that question?
- Examiner
What is the counter metric that might tank if you over optimize that?
- Caller
Countermetrics, you mean?
- Examiner
Yes. A counter metric is something you watch to make sure you are not causing harm or trading off another important outcome while chasing your North Star. What is the counter metric that might tank if you over optimize that?
- Caller
I guess probably the number of users because in that case, we need a lot of outreaching people and customer service people to maintain those relationships.
- Examiner
So your worry is you chase learning gains with heavy touch support, and adoption or retention drops because it does not scale or feels too burdensome for teachers. In your setting, is a false positive or a false negative worse?
- Caller
A false negative?
- Examiner
Walk me through the concrete user impact of that specific failure.
- Caller
Can we end the exam right now?
- Examiner
Yes, we can stop now. Before you go, was the issue time, stress, or that the questions were unclear?
- Caller
Uh...
- Caller
I'm just not ready for the question.
- Examiner
Fair. These are uncomfortable on purpose, like a live demo at TechCrunch Disrupt. Do you want to pause for a moment, or do you want to end the exam?
- Caller
I want to end the exam.
- Examiner
Understood. We will end the exam here. Email the instructor as soon as you can to explain that you ended early and to ask about next steps. Enjoy your break.
- Caller
Thanks
- Examiner
You are welcome.