{ Analytica Model Test_of_Jev_library, encoding="UTF-8" } SoftwareVersion 7.2.3 { System Variables with non-default values: } SampleSize := 1000 TypeChecking := 1 Checking := 1 SaveOptions := 2 SaveValues := 0 {!-50299|DiagramColor Model: 0x00ffffff} {!-50299|DiagramColor Module: 0x00ffffff} {!-50299|DiagramColor LinkModule: 0x00ffffff} {!-50299|DiagramColor Library: 0x00ffffff} {!-50299|DiagramColor LinkLibrary: 0x00ffffff} {!-50299|DiagramColor Form: 0x00ffffff} NodeInfo FormNode: 1,0,0,,0,0,,,,0,,,0 {!-50299|NodeColor Text: 0x00f3f3f3} Model Test_of_Jev_library Title: Test of Jev library Author: Lonnie Chrisman, Ph.D.~ Lumina Decision Systems Date: Fri, Oct 23, 2026 3:28 PM DiagState: 2,0,1,1264,476,17,10 WindState: 2,98,82,720,350 FontStyle: Arial,15 FileInfo: 0,Model Test_of_Jev_library,2,2,0,0,W:\Testmodels\Test of Jev library.ana {!70104|Att_GUID: 5cf950a2-b41b-46a8-b385-a60e885f3fb9} Include 0,LinkLibrary Jev_library,2,2,0,0,Jev library.ana NodeLocation Jev_library: 232,104,1 NodeSize Jev_library: 64,24 Variable Jev_test_call Title: Jev test call Description: Live API smoke test using synthetic text. Evaluating sends the example to TypeSafe and may incur charges. Returns JevResponse; answers->refund->noul is the probability that a refund was requested. Re-evaluation after invalidation makes another API call. Definition: Local request := JevRequest(~ 'Please refund the duplicate charge.',~ JevQuestions(~ refund: JevNoul('Is a refund requested?')~ )~ );~ request -> JevCall() NodeLocation: 232,216,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 ValueState: 2,100,106,416,303,,MIDM Module QA_Testing Title: QA Testing Description: Regression tests for Jev library. Offline tests require no API key. QA Live result sends one synthetic request containing choice, score and noul questions and may incur API charges. Cached results are not a new run; invalidate results before rerunning. Tests do not alter the library or read the API key. NodeLocation: 456,216,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 DiagState: 2,0,1,1250,471,17,10 Index QA_Case Title: QA Case Description: Names of the offline regression cases. Each name selects its assertion in QA_Run_Case. Definition: ['State fields', 'Choice criteria', 'Noul criteria', 'Choice constructor', 'Score constructor', 'Noul defaults', 'Mixed questions', 'Probability fields', 'Legend fields', 'Usage fields', 'Default model mapping', 'Custom model mapping', 'Namespaced model', 'Request top level', 'Noul JSON', 'Noul criteria JSON', 'Choice JSON', 'Score JSON', 'Text escaping', 'Structured state JSON', 'Structured instructions JSON', 'Response noul', 'Response choice', 'Response score zero-based', 'Response usage', 'Response types', 'Response cost', 'Response without cost', 'Response complete', 'Response extra fields', 'Typed answer fields', 'Reject refusal', 'Allow refusal', 'Reject missing answer', 'Allow missing answer', 'Reject empty answers', 'Reject array answers', 'Reject non-JSON', 'Unchecked response', 'Noul probability array', 'Choice probability array', 'Score probability array', 'Reject refusal probability array', 'Reject one choice', 'Reject 256 choices', 'Reject one score level', 'Reject 11 score levels', 'Reject choice type', 'Reject score type', 'Reject noul type', 'Reject empty questions', 'Reject invalid question', 'Accept ten score levels'] NodeLocation: 336,64,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 Variable QA_Fixture_Questions Title: QA Fixture questions Description: The questions that QA Response fixture and QA Refusal fixture answer. Also used by the live test. Definition: JevQuestions(~ refund: JevNoul('Is a refund requested?'),~ intent: JevChoice('Classify the request.', JevChoiceCriteria(refund: 'A refund request', other: 'Anything else')),~ urgency: JevScore('How urgent is the request?', ['Not urgent', 'Urgent'])~ ) NodeLocation: 112,40,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 Variable QA_Response_Fixture Title: QA Response fixture Description: Synthetic System One response body, in TypeSafe's native shape as returned by the gateway's /typesafe/v1/systemone endpoint, including gateway cost metadata. Score levels are zero-based. No API access. Definition: '{"model":"typesafe-ai/jev","answers":{"refund":{"type":"noul","noul":0.95},"intent":{"type":"choice","choice":"refund","confidence":0.8,"probabilities":{"refund":0.8,"other":0.2}},"urgency":{"type":"score","score":0.7,"confidence":0.4,"legend":{"0":"Not urgent","1":"Urgent"},"probabilities":{"0":0.3,"1":0.7}}},"usage":{"input_tokens":12,"output_tokens":6},"provider_metadata":{"gateway":{"cost":"0.00001155","generationId":"gen_test"}}}' NodeLocation: 112,104,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 Variable QA_Refusal_Fixture Title: QA Refusal fixture Description: Synthetic response in which the model declined to answer the urgency question. No API access. Definition: '{"model":"typesafe-ai/jev","answers":{"refund":{"type":"noul","noul":0.9},"intent":{"type":"choice","choice":"other","confidence":0.6,"probabilities":{"refund":0.4,"other":0.6}},"urgency":{"type":"refusal"}},"usage":{"input_tokens":12,"output_tokens":4}}' NodeLocation: 112,168,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 Function QA_Rejected(errText : Text orNull atom;~ errNum : Number atom;~ expected : Text atom;~ expectedNum : optional Number orNull atom = Null) Title: QA Rejected Description: Use in the catch of an expected-rejection test. True when the caught error is the expected one: its ErrorText contains «expected». When ErrorText is Null or empty, as it is for an error raised inside a library function or constructor, falls back to checking that ErrorNumber equals «expectedNum», or by default the number that Error() raises. This separates the library's own rejections from unrelated failures such as type-check or syntax errors. Definition: If Not (IsNull(errText) Or errText = '') Then FindInText(expected, errText) > 0~ Else errNum = (If IsNull(expectedNum) Then Try(Error('QA probe'), catch: ErrorNumber) Else expectedNum) NodeLocation: 112,304,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 Function QA_Run_Case(c : Text atom) Title: QA Run case Description: Runs the offline regression assertion named «c»; returns PASS, FAIL, or ERROR with the error number and any message. Expected-rejection cases force constructor evaluation with TypeOf and check the error with QA_Rejected. Independent cases continue even if another case fails. Definition: Try(~ Local ok :=~ If c = 'State fields' Then (JevState(note: 'hello') -> note) = 'hello'~ Else If c = 'Choice criteria' Then Size(JevChoiceCriteria(a: 'A', b: 'B') -> _members) = 2~ Else If c = 'Noul criteria' Then (JevNoulCriteria('Yes', 'No') -> `false`) = 'No'~ Else If c = 'Choice constructor' Then (JevChoice('Pick', JevChoiceCriteria(a: 'A', b: 'B')) -> type) = 'choice'~ Else If c = 'Score constructor' Then (JevScore('Rate', ['Low', 'High']) -> type) = 'score'~ Else If c = 'Noul defaults' Then IsNull(JevNoul('Question') -> criteria) And (JevNoul('Question') -> type) = 'noul'~ Else If c = 'Mixed questions' Then Size(JevQuestions(a: JevNoul('Q'), b: JevScore('R', ['L','H']), c: JevChoice('C', JevChoiceCriteria(a: 'A', b: 'B'))) -> _members) = 3~ Else If c = 'Probability fields' Then (JevProbabilities(a: 0.25, b: 0.75) -> b) = 0.75~ Else If c = 'Legend fields' Then `->`(ParseJSON('{"0":"Low","1":"High"}', JevLegend), '1') = 'High'~ Else If c = 'Usage fields' Then (JevUsage(12, 6) -> output_tokens) = 6~ Else If c = 'Default model mapping' Then (~ Local r := JevRequest('x', JevQuestions(q: JevNoul('Q')));~ (ParseJSON(r -> JevJSON()) -> model) = 'typesafe-ai/jev'~ ) Else If c = 'Custom model mapping' Then (~ Local r := JevRequest('x', JevQuestions(q: JevNoul('Q')), 'custom');~ (ParseJSON(r -> JevJSON()) -> model) = 'typesafe-ai/custom'~ ) Else If c = 'Namespaced model' Then (~ Local r := JevRequest('x', JevQuestions(q: JevNoul('Q')), 'vendor/model');~ (ParseJSON(r -> JevJSON()) -> model) = 'vendor/model'~ ) Else If c = 'Request top level' Then (~ Local j := ParseJSON(JevRequest('x', JevQuestions(q: JevNoul('Q'))) -> JevJSON());~ Size(j -> _members) = 3 And (j -> state) = 'x' And TypeOf(j -> questions) = '_JSON'~ ) Else If c = 'Noul JSON' Then (~ Local q := ParseJSON(JevRequest('x', JevQuestions(q: JevNoul('Q'))) -> JevJSON()) -> questions -> q;~ (q -> type) = 'noul' And (q -> instructions) = 'Q' And IsNull(q -> criteria)~ ) Else If c = 'Noul criteria JSON' Then (~ Local q := ParseJSON(JevRequest('x', JevQuestions(q: JevNoul('Q', JevNoulCriteria('YES', 'NO')))) -> JevJSON()) -> questions -> q;~ (q -> instructions) = 'Q' And (q -> criteria -> `true`) = 'YES' And (q -> criteria -> `false`) = 'NO'~ ) Else If c = 'Choice JSON' Then (~ Local q := ParseJSON(JevRequest('x', JevQuestions(q: JevChoice('Pick', JevChoiceCriteria(a: 'A', b: Null)))) -> JevJSON()) -> questions -> q;~ Local crit := q -> criteria;~ (q -> type) = 'choice' And (crit -> a) = 'A' And IsNull(crit -> b) And Size(crit -> _members) = 2~ ) Else If c = 'Score JSON' Then (~ Local q := ParseJSON(JevRequest('x', JevQuestions(q: JevScore('Rate', ['Low', 'High']))) -> JevJSON()) -> questions -> q;~ (q -> type) = 'score' And Size(q -> criteria) = 2 And Slice(q -> criteria, 2) = 'High'~ ) Else If c = 'Text escaping' Then (~ Local text := 'Say "hello"' & Chr(10) & 'C:\test';~ Local j := ParseJSON(JevRequest(text, JevQuestions(q: JevNoul(text))) -> JevJSON());~ (j -> state) = text And (j -> questions -> q -> instructions) = text~ ) Else If c = 'Structured state JSON' Then (~ Local j := ParseJSON(JevRequest(JevState(amount: 12, text: 'refund'), JevQuestions(q: JevNoul('Q'))) -> JevJSON());~ (j -> state -> amount) = 12 And (j -> state -> text) = 'refund'~ ) Else If c = 'Structured instructions JSON' Then (~ Local q := ParseJSON(JevRequest('x', JevQuestions(q: JevNoul(_Struct(task: 'refund')))) -> JevJSON()) -> questions -> q;~ (q -> instructions -> task) = 'refund'~ ) Else If c = 'Response noul' Then (~ Local a := JevParseResponse(QA_Response_Fixture, QA_Fixture_Questions) -> answers -> refund;~ TypeOf(a) = 'JevAnswer' And (a -> type) = 'noul' And (a -> noul) = 0.95~ ) Else If c = 'Response choice' Then (~ Local a := JevParseResponse(QA_Response_Fixture, QA_Fixture_Questions) -> answers -> intent;~ (a -> choice) = 'refund' And (a -> confidence) = 0.8 And (a -> probabilities -> other) = 0.2~ ) Else If c = 'Response score zero-based' Then (~ Local a := JevParseResponse(QA_Response_Fixture, QA_Fixture_Questions) -> answers -> urgency;~ (a -> score) = 0.7 And (a -> confidence) = 0.4 And~ `->`(a -> legend, '0') = 'Not urgent' And `->`(a -> probabilities, '1') = 0.7~ ) Else If c = 'Response usage' Then (~ Local r := JevParseResponse(QA_Response_Fixture, QA_Fixture_Questions);~ (r -> usage -> input_tokens) = 12 And (r -> usage -> output_tokens) = 6 And (r -> model) = 'typesafe-ai/jev'~ ) Else If c = 'Response types' Then (~ Local r := JevParseResponse(QA_Response_Fixture, QA_Fixture_Questions);~ TypeOf(r) = 'JevResponse' And TypeOf(r -> answers) = 'JevAnswers' And TypeOf(r -> usage) = 'JevUsage'~ ) Else If c = 'Response cost' Then~ Abs((JevParseResponse(QA_Response_Fixture, QA_Fixture_Questions) -> cost) - 0.00001155) < 1e-12~ Else If c = 'Response without cost' Then~ IsNull(JevParseResponse('{"model":"m","answers":{"q":{"type":"noul","noul":0.5}},"usage":{"input_tokens":1,"output_tokens":0}}') -> cost)~ Else If c = 'Response complete' Then~ Size(JevParseResponse(QA_Response_Fixture, QA_Fixture_Questions) -> unanswered) = 0~ Else If c = 'Response extra fields' Then (~ Local pm := JevParseResponse(QA_Response_Fixture, QA_Fixture_Questions) -> provider_metadata;~ (pm -> gateway -> generationId) = 'gen_test'~ ) Else If c = 'Typed answer fields' Then (~ Local a := JevAnswer('choice', choice: 'a', probabilities: _Struct(a: 0.8, b: 0.2), legend: _Struct(a: 'A'));~ TypeOf(a -> probabilities) = 'JevProbabilities' And TypeOf(a -> legend) = 'JevLegend' And IsNull(a -> score)~ ) Else If c = 'Reject refusal' Then~ Try(TypeOf(JevParseResponse(QA_Refusal_Fixture, QA_Fixture_Questions)) = 'REJECTED',~ catch: QA_Rejected(ErrorText, ErrorNumber, 'urgency (refusal)'))~ Else If c = 'Allow refusal' Then (~ Local r := JevParseResponse(QA_Refusal_Fixture, QA_Fixture_Questions, allowIncomplete: True);~ JoinText(r -> unanswered, separator: ',') = 'urgency' And~ (r -> answers -> urgency -> type) = 'refusal' And (r -> answers -> refund -> noul) = 0.9~ ) Else If c = 'Reject missing answer' Then~ Try(TypeOf(JevParseResponse('{"model":"m","answers":{"refund":{"type":"noul","noul":0.9}},"usage":{"input_tokens":1,"output_tokens":0}}', QA_Fixture_Questions)) = 'REJECTED',~ catch: QA_Rejected(ErrorText, ErrorNumber, 'intent (missing)'))~ Else If c = 'Allow missing answer' Then (~ Local r := JevParseResponse('{"model":"m","answers":{"refund":{"type":"noul","noul":0.9}},"usage":{"input_tokens":1,"output_tokens":0}}', QA_Fixture_Questions, allowIncomplete: True);~ Local u := r -> unanswered;~ Size(u) = 2 And Size(Subset(u = 'intent')) = 1 And Size(Subset(u = 'urgency')) = 1 And (r -> answers -> refund -> noul) = 0.9~ ) Else If c = 'Reject empty answers' Then~ Try(TypeOf(JevParseResponse('{"model":"m","answers":{},"usage":{"input_tokens":1,"output_tokens":0}}')) = 'REJECTED',~ catch: QA_Rejected(ErrorText, ErrorNumber, 'has no answers.'))~ Else If c = 'Reject array answers' Then~ Try(TypeOf(JevParseResponse('{"model":"m","answers":[{"type":"predicate","name":"q","probability":0.5}],"usage":{"input_tokens":1,"output_tokens":0}}')) = 'REJECTED',~ catch: QA_Rejected(ErrorText, ErrorNumber, 'no answers map'))~ Else If c = 'Reject non-JSON' Then~ Try(TypeOf(JevParseResponse(Chr(60) & 'html>Bad gateway' & Chr(60) & '/html>')) = 'REJECTED',~ catch: QA_Rejected(ErrorText, ErrorNumber, 'JSON', Try(ParseJSON(Chr(60) & 'html>'), catch: ErrorNumber)))~ Else If c = 'Unchecked response' Then~ IsNull(JevParseResponse(QA_Response_Fixture) -> unanswered)~ Else If c = 'Noul probability array' Then (~ Local x := JevProbabilityArray(JevParseResponse(QA_Response_Fixture) -> answers -> refund);~ Size(x) = 2 And Abs(Sum(x, x.Outcome) - 1) < 1e-9 And x[x.Outcome = 'true'] = 0.95~ ) Else If c = 'Choice probability array' Then (~ Local x := JevProbabilityArray(JevParseResponse(QA_Response_Fixture) -> answers -> intent);~ Size(x) = 2 And x[x.Outcome = 'other'] = 0.2 And x[x.Outcome = 'refund'] = 0.8~ ) Else If c = 'Score probability array' Then (~ Local r := JevParseResponse('{"model":"m","answers":{"u":{"type":"score","score":1.2,"confidence":0.4,"legend":{"0":"a","1":"b","2":"c"},"probabilities":{"2":0.4,"0":0.2,"1":0.4}}},"usage":{"input_tokens":1,"output_tokens":0}}');~ Local x := JevProbabilityArray(r -> answers -> u);~ Size(x) = 3 And Slice(x, x.Outcome, 1) = 0.2 And Slice(x.Outcome, 1) = 0 And x[x.Outcome = 2] = 0.4~ ) Else If c = 'Reject refusal probability array' Then~ Try(TypeOf(JevProbabilityArray(JevAnswer('refusal'))) = 'REJECTED',~ catch: QA_Rejected(ErrorText, ErrorNumber, 'has no probabilities'))~ Else If c = 'Reject one choice' Then~ Try(TypeOf(JevChoice('Q', JevChoiceCriteria(a: 'A'))) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, '2 to 255 options'))~ Else If c = 'Reject 256 choices' Then (~ Local parts := For k := 1..256 Do MakeJSON('o' & k, flags: 8) & ': "option"';~ Local crit := ParseJSON('{' & JoinText(parts, separator: ',') & '}', JevChoiceCriteria);~ Try(TypeOf(JevChoice('Q', crit)) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, '2 to 255 options'))~ ) Else If c = 'Reject one score level' Then~ Try(TypeOf(JevScore('Q', ['one'])) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, '2 to 10 levels'))~ Else If c = 'Reject 11 score levels' Then~ Try(TypeOf(JevScore('Q', 1..11)) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, '2 to 10 levels'))~ Else If c = 'Reject choice type' Then~ Try(TypeOf(JevChoice('Q', JevChoiceCriteria(a: 'A', b: 'B'), type: 'bad')) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, 'type must be choice'))~ Else If c = 'Reject score type' Then~ Try(TypeOf(JevScore('Q', ['L','H'], type: 'bad')) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, 'type must be score'))~ Else If c = 'Reject noul type' Then~ Try(TypeOf(JevNoul('Q', type: 'bad')) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, 'type must be noul'))~ Else If c = 'Reject empty questions' Then~ Try(TypeOf(JevQuestions()) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, 'at least one named question'))~ Else If c = 'Reject invalid question' Then~ Try(TypeOf(JevQuestions(q: 'invalid')) = 'REJECTED', catch: QA_Rejected(ErrorText, ErrorNumber, 'must be JevChoice, JevScore, or JevNoul'))~ Else If c = 'Accept ten score levels' Then Size(JevScore('Q', 1..10) -> criteria) = 10~ Else Error('Unknown QA case: ' & c);~ If IsNull(ok) Then 'FAIL: Null assertion'~ Else If ok Then 'PASS' Else 'FAIL',~ catch: 'ERROR (' & ErrorNumber & ')' & (If IsNull(ErrorText) Or ErrorText = '' Then '' Else ': ' & ErrorText)~ ) NodeLocation: 112,240,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 Variable QA_Offline_Results Title: QA Offline results Description: Deterministic regression cases; PASS, FAIL, or ERROR for each case. No API calls. Definition: QA_Run_Case(QA_Case) NodeLocation: 336,192,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 Variable QA_Live_Result Title: QA Live result Description: LIVE: one synthetic API request with the three QA Fixture questions. May incur charges when evaluated after invalidation. Validates response types, numeric bounds, probability normalization, completeness, usage and cost, not exact probabilistic predictions. Returns status, details, and full response. Definition: Try(~ Local request := JevRequest('Please refund the duplicate charge. This is urgent.', QA_Fixture_Questions);~ Local r := request -> JevCall();~ Local a := r -> answers;~ Local p := a -> refund -> noul;~ Local c := a -> intent;~ Local s := a -> urgency;~ Local bounds := p >= 0 And p <= 1 And~ (c -> confidence) >= 0 And (c -> confidence) <= 1 And~ (s -> confidence) >= 0 And (s -> confidence) <= 1 And~ (s -> score) >= 0 And (s -> score) <= 1;~ Local cp := JevProbabilityArray(c);~ Local sp := JevProbabilityArray(s);~ Local normalized := Abs(Sum(cp, cp.Outcome) - 1) < 0.001 And Abs(Sum(sp, sp.Outcome) - 1) < 0.001 And~ Min(cp, cp.Outcome) >= 0 And Max(cp, cp.Outcome) <= 1 And Size(cp) = 2 And Size(sp) = 2;~ Local types := TypeOf(r) = 'JevResponse' And TypeOf(a) = 'JevAnswers' And~ (a -> refund -> type) = 'noul' And (c -> type) = 'choice' And (s -> type) = 'score';~ Local choiceValid := (c -> choice) = 'refund' Or (c -> choice) = 'other';~ Local complete := Size(r -> unanswered) = 0;~ Local usageValid := (r -> usage -> input_tokens) >= 0 And (r -> usage -> output_tokens) >= 0;~ Local costValid := IsNull(r -> cost) Or (r -> cost) >= 0;~ Local ok := types And bounds And normalized And choiceValid And complete And usageValid And costValid;~ _Struct(~ status: If IsNull(ok) Then 'FAIL' Else If ok Then 'PASS' Else 'FAIL',~ types: types,~ bounds: bounds,~ probabilities_normalized: normalized,~ choice_valid: choiceValid,~ complete: complete,~ usage: usageValid,~ cost: r -> cost,~ response: r~ ),~ catch: _Struct(status: 'ERROR', error_number: ErrorNumber, error_text: ErrorText)~ ) NodeLocation: 336,304,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 Variable QA_Summary Title: QA Summary Description: Evaluating runs any uncached offline cases AND the live test. Live test may incur API charges. Offline counts are per QA Case; live is one combined integration request. Definition: Local passed := Sum(QA_Offline_Results = 'PASS', QA_Case);~ Local total := IndexLength(QA_Case);~ Local live := QA_Live_Result -> status;~ _Struct(~ overall: If passed = total And live = 'PASS' Then 'PASS' Else 'FAIL',~ offline_passed: passed,~ offline_total: total,~ offline_not_passed: total - passed,~ live_status: live~ ) NodeLocation: 560,248,2 NodeSize: 80,24 NodeInfo: 1,,,,,,,0,,,,,,0 ValueState: 2,36,42,400,600,0,MIDM Close QA_Testing Close Test_of_Jev_library