Recording, examiner and caller mixed. Download the MP3.
What the caller tried
- gave a random ID and own first name first
- switched to the prefilled persona after rejection
- pitched a real-estate search assistant
- answered metric questions briefly
Transcript
Speech to text, so expect misheard words; the recording is the record. Click a time to play from there.
- Examiner
Hey there, I'm professor Foster Provost from NYU Stern. I am ready to conduct your exam today. Am I speaking to [DEMO STUDENT]? Can you please tell me your name and your NET ID?
- Caller
try, 7898
- Examiner
I need you to state your name and your NetID exactly, and I will verify it. What is your name?
- Caller
The name is Marco.
- Examiner
Marco, what is your NetID?
- Caller
0000
- Examiner
That NetID does not match the one I have on file, which is [DEMO NETID]. Please hang up and email the instructor immediately to resolve the identity mismatch.
- Caller
The name is [DEMO STUDENT].
- Examiner
What is your NetID?
- Caller
[DEMO NETID].
- Examiner
Great, hello [DEMO STUDENT]. I see you worked on Team B. Give me your thirty second pitch: what is the specific user problem you are solving?
- Caller
We did try to implement a thing to talk to Zillow.
- Examiner
Hold on. What is the specific user problem you are solving?
- Caller
Trying to have a faster experience with Zillow.
- Examiner
And why does that specific problem require AI?
- Caller
Because we felt like it was easier than browsing lots of stuff.
- Examiner
What is your North Star metric for this product?
- Caller
We used time to find the appropriate match.
- Examiner
Good. Now, what is the counter metric that might tank if you over optimize that?
- Caller
That just by randomly clicking and hoping to find a good spot might do the trick, but I feel it's difficult to cheat that metric.
- Examiner
You are not answering the question. What is the counter metric that might get worse if you push time to match down hard?
- Caller
I don't know.
- Examiner
Pick just one plausible counter metric. What would you choose?
- Caller
Accuracy.
- Examiner
In your setting, is a false positive or a false negative worse?
- Caller
I think a false positive is worse.
- Examiner
Walk me through the concrete user impact of that specific failure.
- Caller
Is that I think that the user found what he was looking for, where instead he was not.
- Examiner
Pick one F A T P issue relevant to your domain.