autodetect fails to restore categorizer

[valeriy42]

Recently we observe bursts for error with the following logs:

autodetect/283 process stopped unexpectedly: JSON state must be object at root10 Cannot restore categorizer - a element expected but found = JSON restore failed Failed to restore state ML anomaly detector job failed
[autodetect/192] [CCmdSkeleton.cc@49] Failed to restore state
[autodetect/192] [CJsonStateRestoreTraverser.cc@317] JSON state must be object at root10

It's unclear what causes this error. The first step is to enhance logging to get more information for further investigation.

[valeriy42]

I added #2867 for enhanced logging.

[valeriy42]

There was a new failure with the following log message:

autodetect/512 process stopped unexpectedly:
JSON state must be object at root. Found token type: array_end (value: 10), isArrayOfObjects: true, isEof: false, stream state - bad: false, fail: true, eof: true, bytes remaining: 0, buffer position: 2
Cannot restore categorizer - a element expected but found =
JSON restore failed
Failed to restore state
ML anomaly detector job failed

[edsavage]

I've managed to reproduce this with the following steps in the dev console.

• Create Job

PUT /_ml/anomaly_detectors/logs_services_count_logs_categories
{
  "job_id": "logs_services_count_logs_categories",
  "groups": [
    "apm",
    "logs"
  ],
  "description": "",
  "analysis_config": {
    "bucket_span": "15m",
    "categorization_field_name": "message",
    "categorization_analyzer": {
      "char_filter": [
        "first_line_with_letters"
      ],
      "tokenizer": "ml_standard",
      "filter": [
        {
          "type": "stop",
          "stopwords": [
            "Monday",
            "Tuesday",
            "Wednesday",
            "Thursday",
            "Friday",
            "Saturday",
            "Sunday",
            "Mon",
            "Tue",
            "Wed",
            "Thu",
            "Fri",
            "Sat",
            "Sun",
            "January",
            "February",
            "March",
            "April",
            "May",
            "June",
            "July",
            "August",
            "September",
            "October",
            "November",
            "December",
            "Jan",
            "Feb",
            "Mar",
            "Apr",
            "May",
            "Jun",
            "Jul",
            "Aug",
            "Sep",
            "Oct",
            "Nov",
            "Dec",
            "GMT",
            "UTC"
          ]
        },
        {
          "type": "limit",
          "max_token_count": "100"
        }
      ]
    },
    "per_partition_categorization": {
      "enabled": true,
      "stop_on_warn": false
    },
    "detectors": [
      {
        "detector_description": "count by mlcategory partitionfield=\"service.name\"",
        "function": "count",
        "by_field_name": "mlcategory",
        "partition_field_name": "service.name",
        "detector_index": 0
      }
    ],
    "influencers": [
      "mlcategory",
      "service.name"
    ],
    "model_prune_window": "30d"
  },
  "analysis_limits": {
    "model_memory_limit": "1304mb",
    "categorization_examples_limit": 4
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_plot_config": {
    "enabled": false,
    "annotations_enabled": false
  },
  "model_snapshot_retention_days": 10,
  "daily_model_snapshot_retention_after_days": 1,
  "results_index_name": "shared",
  "allow_lazy_open": false,
    "datafeed_config": {
    "datafeed_id": "datafeed-logs_services_count_logs_categories",
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "indices": [
      "cat_job_index"
    ]
  }
}

• Create datafeed index

DELETE /cat_job_index
PUT /cat_job_index
PUT cat_job_index/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "message": {
      "type": "text"
    },
    "service.name": {
      "type": "text"
    }
  }
}

• Put a document with an empty message field into the index

POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "",
  "service.name": "nginx"
}

• Start the job as lookback.
• After the job closes, add more data to the index

POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "blah blah foo bar",
  "service.name": "nginx"
}
POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "blah blah foo bar baz",
  "service.name": "nginx"
}
POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "blah blah foo bar rab",
  "service.name": "nginx"
}
POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "blah blah foo bar baz",
  "service.name": "nginx"
}
POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "blah blah foo bar",
  "service.name": "nginx"
}
POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "blah blah foo bar",
  "service.name": "nginx"
}
POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "blah blah foo bar",
  "service.name": "nginx"
}
POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "blah blah foo bar",
  "service.name": "nginx"
}
POST /cat_job_index/_doc
{
  "@timestamp": 1768622400,
  "message": "",
  "service.name": "nginx"
}

• Start the job in lookback again
• The job goes into the failed state.
• Check the logs

[2026-01-28T15:59:17,174][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] Opening job [logs_services_count_logs_categories]
[2026-01-28T15:59:17,178][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
[2026-01-28T15:59:17,281][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19394] [CResourceMonitor.cc@85] Setting model memory limit to 1304 MB
[2026-01-28T15:59:17,553][INFO ][o.e.x.m.d.DatafeedJob    ] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Datafeed started (from: 1970-01-01T00:00:00.000Z to: 2026-01-28T02:59:12.365Z) with frequency [450000ms]
[2026-01-28T15:59:17,556][WARN ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19394] [CFieldDataCategorizer.cc@178] Assigning ML category -1 to record with blank message field:
@timestamp,message,mlcategory,...,service.name,.
1768622,,,,,
[2026-01-28T15:59:17,592][INFO ][o.e.x.m.d.DatafeedJob    ] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Lookback has finished
[2026-01-28T15:59:17,592][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] attempt to stop datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories]
[2026-01-28T15:59:17,592][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] try lock [20s] to stop datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories]...
[2026-01-28T15:59:17,593][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] stopping datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories], acquired [true]...
[2026-01-28T15:59:17,720][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories] has been stopped
[2026-01-28T15:59:17,858][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] Closing job [logs_services_count_logs_categories], because [close job (api)]
[2026-01-28T15:59:17,859][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19394] [CCmdSkeleton.cc@66] Handled 1 records
[2026-01-28T15:59:17,859][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19394] [CAnomalyJob.cc@1646] Pruning obsolete models
[2026-01-28T15:59:17,859][WARN ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19394] [CFieldDataCategorizer.cc@447] No partition-specific categorizers found
[2026-01-28T15:59:17,903][INFO ][o.e.x.m.p.AbstractNativeProcess] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] State output finished
[2026-01-28T15:59:17,991][INFO ][o.e.x.m.j.p.a.o.AutodetectResultProcessor] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] 0 buckets parsed from autodetect output
[2026-01-28T15:59:18,045][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] autodetect connection for job closed
[2026-01-28T16:04:55,237][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] Opening job [logs_services_count_logs_categories]
[2026-01-28T16:04:55,241][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Loading model snapshot [1769569157] with latest_record_timestamp [1970-01-21T11:17:02.000Z], job latest_record_timestamp [1970-01-21T11:17:02.400Z]
[2026-01-28T16:04:55,248][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] [Ed’s-MacBook-Pro-M4] Restoring quantiles for job 'logs_services_count_logs_categories'
[2026-01-28T16:04:55,395][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19449] [CResourceMonitor.cc@85] Setting model memory limit to 1304 MB
[2026-01-28T16:04:55,399][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19449] [CAnomalyJob.cc@936] Processing is already complete to time 1768500
[2026-01-28T16:04:55,406][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19449] [CStateDecompressor.cc@138] Failed to find valid JSON
[2026-01-28T16:04:55,406][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19449] [CJsonStateRestoreTraverser.cc@370] JSON state must be object at root. Found token type: array_end (value: 10), isArrayOfObjects: true, isEof: false, stream state - bad: false, fail: true, eof: true, bytes remaining: 0, buffer position: 2
[2026-01-28T16:04:55,406][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19449] [CFieldDataCategorizer.cc@349] Cannot restore categorizer - a element expected but found =
[2026-01-28T16:04:55,406][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19449] [CFieldDataCategorizer.cc@313] JSON restore failed
[2026-01-28T16:04:55,406][FATAL][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19449] [CCmdSkeleton.cc@49] Failed to restore state
[2026-01-28T16:04:55,406][FATAL][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/19449] [Main.cc@333] ML anomaly detector job failed
[2026-01-28T16:04:55,406][INFO ][o.e.x.m.p.AbstractNativeProcess] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] State output finished
[2026-01-28T16:04:55,406][INFO ][o.e.x.m.j.p.a.o.AutodetectResultProcessor] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] 0 buckets parsed from autodetect output
[2026-01-28T16:04:55,407][ERROR][o.e.x.m.p.AbstractNativeProcess] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] autodetect/19449 process stopped unexpectedly: JSON state must be object at root. Found token type: array_end (value: 10), isArrayOfObjects: true, isEof: false, stream state - bad: false, fail: true, eof: true, bytes remaining: 0, buffer position: 2
Cannot restore categorizer - a element expected but found =
JSON restore failed
Failed to restore state
ML anomaly detector job failed

[2026-01-28T16:04:55,456][WARN ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [controller/18916] [CDetachedProcessSpawner.cc@212] Child process with PID 19449 has exited with exit code 1
[2026-01-28T16:04:55,514][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] Successfully set job state to [failed] for job [logs_services_count_logs_categories]

• Check the state documents

GET /.ml-state-000001/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ml-state-000001",
        "_id": "logs_services_count_logs_categories_model_state_1769569157#1",
        "_score": 1,
        "_source": {
          "compressed": [
            "H4sIAAAAAAAA",
            "/+2b624cuRGF/+cpgvktCyzeqZchpNFIstcXwOsECAS9e77D7h71jMaOlKwX8Ga0u1ZPd5Osyzmnihzv4+Z6c7WxkmtybnOxueFTiFz8trl6HM90d6vfV/Pzx025NOMOH+/nF5YXb54HjNGP852neeQtn1peBly589X56nz1y109rSXhdroa17uZ/3fz70UfHvj9LmazXLOzGn0JtTQnoXn//Ucfvv/oo9REq9llaSW3YCHW7IPFsguubp6ehmGHIvQ0DDwhTNPFnW7nyzBJ2/4l27906Xx048cvJhwI39Va+vTxYv5nLP38YJ7nUC83+1huZjOR2b0pGuTa9FPqlT7lFvWTFlU+WnNZeZ+q5eMr5rZQ9dN+wtS+jJ/6E6auYfyknzC3/8OjbL+CkXxKE+BdeevcBxIwy8PDfA/OPz4Nfj9O9w6ZdaKb2NghO2Y6P+xp9361zIfVNX3M5u764+873pcm7L7ttt++fNXK27kjeZwGHy1sK/naTgHS+tsv//j8bbZhFrpnsRnvLXG9mwVkCtmc702YlwmrZfQ5zp9TboN3upeO3rFl0BIkX0+M8u1oWPDLjRisZcubtRpFC2S386fVXK367luosTXvjcvkPDdL7fyRMtch95Ksch19R4KKs9xy916Te1esI0wum0ule2u1phZi9y7nkkssvgfWqHqn9lgTd1ii+xBoQUutoRtzJNQnhh4iql9carW3ZN7QvNIDap9SDZlFm0f2vUvWrVADSkyVtVqlELjquJtYCWOC78YMruAEd6urPCk5MSyE2grrdewPvgbve/TOp0ChiT0Fl1NtuGrFYjWn54HHXsGIPYToU+XlzAKhuBAIXvc54r5Pxlo5Rd+aa06Gg3JsD6zAsoWYN3ykfiVXWuwUsOybM4cLMfiQWa0T5oxbKWIsN4IrXukyVk6YnrCmNdLK2yTB1ZwKk3RPrDN2+dIJDCEMBLFkSwVbsZsLzPY85netDhj0EpzqqJz1scVEwgWGYkYOWAg0ENqYG8Eg3cRGOXLeckkM61SNWGuOVnrEC0M7WJRQABAszCADO0ldIkIgifcNYBEzZohEU64loav76sAV1RwLQsmulVCARpINXFonPT4T5oYxnjhUXxIWFCUuJuMyxqZ9TSXEqbgGXhkWPWknVKQDAOIw8CWhCmwMmWiQeiLO0njeUmZaQTJ7TZCV8OKALFUGIBZe01CsBRtMELg0M0fyHd6yYgFoTOUaQYFQwhbcAtOMAtDWeINwKIMEGQyQF3DXyB4G0NDwqTpeTcwAHAR0iOpFOYsZ6jaZChuqb6VBPsN9PCCEmJmL83qTQAcUXXjEZo3GFaWNJ5n5yb63kijOwME5aEgoAtGNoFyLupB510eRB8SCRyfkkTRIocxkq2YkGgMEgmC5ow3IRyO8zcSrIkJCAVJfFHRPpojPAE7wKQpN/OEH5QAz/rXKGko2OK4uySjMwKeCegQ4wb/kTWgwNX+CZm3KCrYrlzXxr2AMyDJZJC2J9sZn2CxRUlo9dkmESsyKdRJJ9LEn4NlIKqtC61KiOGZkrJGPKCXyLEV3KqUjlqiSxxZED5wG643YFuhuvZLoRCzEJ1EQqDhxgNyLkH2omCgg4hLqxG8SxyAoS/RxOAIWxgDfSgKjyATDYmFIKgivC1G0kxfoIDnHcVaJiJwirwKLhDBLKUpwQ8VbHKqKdyM7qC1xUjmIijS3QHEhh4IHik+kvVCF3srZLjMQjsRiFAzEi2csBqoJLzRlmJPwKFQoBwmi8SNSkI5BiBiRihFeN/JeqQ/CA3JQRHDvAKjqRRlJxy4CCFJa4hdQQDdSy8kPzvhAKlV+hKSEh16YRrUDj4BpBUAhq3IREd6CiNQMuKfEF+EcUEMNQ10QtjCwCQS0IgnARxYgDWSzUoOYAPtRdqjekTsnAVWFaFJFyEsQSAKgrTITWlbEJiAIoC7XYQAyHSFXkGY7i5KwPoS0uSIlDxAD1qB/qmxBBbHJEwZiI8bkjH5SaCVZACvVodbMnoEC7CCaJoAK+zig0iUlpL7k4nPoSY+R9dSpZYTRS1SVZ4Rk4FRERAuZvgyOkPFA2SECccArUgGpouAggHMKVBan0Sgky+uFIHAhRLqiSCiNAC6HkqSCRJj/YHFAZUvO0hNtOkA8PhG1DGNJLpHMDjprIQY7eUjUsAkSql8Qrofmo+4mPHrRh0JB8WhRGUDs8FsyACC5TwTlKxFmRQUI9meKCZQy5Diqc6EbAckQVYWfFFKisI/+gd7FkZ84oFBGfCibsMvLFyKUqTsCA2hRuFAk1MoJikxLN8QiKpYyHgujKAHzBYzCuyELhl5dWNapXAQtdA+AOqqqV2UQKZSDTq2Nl13q3AptU1bHpOZDEKTvcGHmN/gVmWuHO550FJGPeoPUjAAVkBSUVgkNCfFCIIkS52ESwXFSfbUdcI3Iif2kBCwbkk9GuAC26pTYk4I09UdMi7+dageO0EYMwQaVeqZ3EmnqtvBFUDK5IGhE1IP30dqp5rcqmWUE0iFU0k3ZeHl0U1BXvSExgR1JvQCDWdVNtI/KFr73CpZpZMgbZRafitS3ySjeVptZ6Sbogomqiih9JrFGMiGGlE0cZTiVSxiu0mfVDOIE6NAqFJjwg1zaZOp8RnRhEgboIRQmgxhHXzVKWY6SR64IOuUzizZRxUXdBJMKn5IffKEoEJZBEhQeJR2iSXkEYwmIQFXCQX/XxcusumEyFohB4zRyDKBHNAw8Q1R1x1SjCCkIN2BgpqB+kp0esALENN40biZBVAAa1YK4JbpvpzKuZgnQ5IExZK9UGaO6ABjAv2LY1BsTf5AbFBDwYKr5sVXJOrEO1C2VTVqKhOklqiUGzG000sAGNJIZRiAa2mSIuiwKttS0J4lcUg8FwUEUZXGSIVdRXbQPcDL1cJHK4MpopOlk1JZiFq0xPNOuBoDTwhADsAWFs7RGjYOpYSJubBvovZAqUIDndFZJLzCYkoafOK49Agwcoq0zqKw2D2u0Iag22jiyB3JJDRVe8kj1zeAXj8ABeqOCrrKABGAXKadjoMNS2y8iigxir9oObTa0Cyowlo6Nu7CETYcDS2hhswH7pKqQJUdCMuZLnrEL+ql1U3IpNAR2eEPzCGeKmm3ipTChinKa/Kn/YJMTx4YFv5okgSj0DF+cdGn04iwFQwlyUk1qKgBQwmSBZENqqAZbZgk7Kmx0YQgn/d3UNGhrSnKj1ApctgEp0IWnxLuq6tGPjc32sqs+OH+bPiz78+tpf34z/Vo25tvlUG91b7eebD7yU+Vw8/HB4/4cQs+1/McXNw823MtHf/gxrGb4dDzDfLbxNH6evyw5PGBcThdvlxPL25cj//MBx3IasD6fXR1wfPq4vf62u//y9V+Hpxy/777+8/12d/n5+tNuPso5ePXh+J3TpyIvU7eyabr5zv60xB1+2M0Tr3M53zy++8Zx/r8cF9487q3gOjrCnsP7bjluGxdHgX7++u97B3W7ZdG93w+ro6/lZM3GIaiwxf6axgEFo6uljtEQhHSFFEqQ1Ya1qH2S2xP9dnW69fx9xMn7t9+5vzt5/2llbvkFzF2+tTjKy/lr2fPV+er/7ur8tezm/LXs+WvZ89eyv9zXsvezbhy6fdSgblfSMD/6MMfG+Skgtrfz0uYILZsJ2/xIHZfW7lkdn/bN4A+MuNlvrk4ZYZvTUmIvJPlVi92u8nLC41dk5hWLDLW9P7XIMtnp8K5mes0y2/Xzx30T+9yl/i9+rrra03nZbzf3O+lpTzkhcXT+J5H126puzuA+TvUhNbfLaqu0L38f4Hkvdgb+nwD8m5XjJ5GxoPfiR/B/61SfDlx7ATy98vkMgbP2nbXvDPy/mvY9TKmdN253M9bZGvo4dbK7d+nYj4t9BC7csS+rFvYPmWk6DX2/T8DqeG3G6eznQJL+Ts0UAuFv6l3nY+bbvcs3P8yWZxsca8hx7tr3h9knkjaB4/7+6+7+ej7Ov57FY/P3Q5ofnCe++F5gdXb4HL3bdZLG//6RvV+Yvxmb6P3/FPL0t38DXD3hXCUy",
            "AAA="
          ],
          "eos": true
        }
      },
      {
        "_index": ".ml-state-000001",
        "_id": "logs_services_count_logs_categories_quantiles",
        "_score": 1,
        "_source": {
          "job_id": "logs_services_count_logs_categories",
          "timestamp": 0,
          "quantile_state": "H4sIAAAAAAAA/52MQQ7CIBBF9x5j1o2hWDXt1mM0LrCMFW2hAjVpmt7dgdLEtSwY/nt/qGfou2ZEqMBN7vIQukXIiL1wIiYGFdN7FNqrDp1E11g1eGU0aWuMj/6D1q3oQNmrPnzI6CmgmsMFBS+L8nTm5ZHoLVmV5jNNubU5y39qzYZZgvkKKS9LBvjX1j0u7Fk8ZNpQl0kvu+sXq6rQyRoBAAA="
        }
      }
    ]
  }
}

• Note that from the above output there is no categorization state document.

I think that this confirms earlier speculation that the error was caused by an empty/missing categorization state document. The categorizer code expects some kind of state document to be restored from and errors when one is not found on the stream.

[edsavage]

Repeating the steps above using a build including #2895 shows that the situation is now handled more gracefully...

[2026-01-28T16:34:25,205][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] Opening job [logs_services_count_logs_categories]
[2026-01-28T16:34:25,216][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
[2026-01-28T16:34:26,176][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20153] [CResourceMonitor.cc@85] Setting model memory limit to 1304 MB
[2026-01-28T16:34:26,368][INFO ][o.e.x.m.d.DatafeedJob    ] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Datafeed started (from: 1970-01-01T00:00:00.000Z to: 2026-01-28T03:34:22.630Z) with frequency [450000ms]
[2026-01-28T16:34:26,383][WARN ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20153] [CFieldDataCategorizer.cc@178] Assigning ML category -1 to record with blank message field:
@timestamp,message,mlcategory,...,service.name,.
1768622,,,,,
[2026-01-28T16:34:26,421][INFO ][o.e.x.m.d.DatafeedJob    ] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Lookback has finished
[2026-01-28T16:34:26,421][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] attempt to stop datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories]
[2026-01-28T16:34:26,421][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] try lock [20s] to stop datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories]...
[2026-01-28T16:34:26,421][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] stopping datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories], acquired [true]...
[2026-01-28T16:34:26,533][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories] has been stopped
[2026-01-28T16:34:26,822][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] Closing job [logs_services_count_logs_categories], because [close job (api)]
[2026-01-28T16:34:26,823][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20153] [CCmdSkeleton.cc@66] Handled 1 records
[2026-01-28T16:34:26,823][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20153] [CAnomalyJob.cc@1646] Pruning obsolete models
[2026-01-28T16:34:26,825][WARN ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20153] [CFieldDataCategorizer.cc@453] No partition-specific categorizers found
[2026-01-28T16:34:26,886][INFO ][o.e.x.m.p.AbstractNativeProcess] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] State output finished
[2026-01-28T16:34:26,962][INFO ][o.e.x.m.j.p.a.o.AutodetectResultProcessor] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] 0 buckets parsed from autodetect output
[2026-01-28T16:34:27,038][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] autodetect connection for job closed
[2026-01-28T16:36:43,289][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] Opening job [logs_services_count_logs_categories]
[2026-01-28T16:36:43,294][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Loading model snapshot [1769571266] with latest_record_timestamp [1970-01-21T11:17:02.000Z], job latest_record_timestamp [1970-01-21T11:17:02.400Z]
[2026-01-28T16:36:43,296][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] [Ed’s-MacBook-Pro-M4] Restoring quantiles for job 'logs_services_count_logs_categories'
[2026-01-28T16:36:43,462][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20172] [CResourceMonitor.cc@85] Setting model memory limit to 1304 MB
[2026-01-28T16:36:43,465][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20172] [CAnomalyJob.cc@936] Processing is already complete to time 1768500
[2026-01-28T16:36:43,471][WARN ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20172] [CStateDecompressor.cc@126] Error parsing JSON: "". Encountered NULL character in stream before parsing has started.
[2026-01-28T16:36:43,471][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20172] [CStateDecompressor.cc@152] Failed to find valid JSON
[2026-01-28T16:36:43,471][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20172] [CFieldDataCategorizer.cc@338] Expected categorizer persisted state but no state exists
[2026-01-28T16:36:43,472][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20172] [CFieldDataCategorizer.cc@287] JSON restore failed
[2026-01-28T16:36:43,472][WARN ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20172] [CFieldDataCategorizer.cc@644] Discarding corrupt categorizer state - will re-categorize from scratch
[2026-01-28T16:36:43,703][INFO ][o.e.x.m.d.DatafeedJob    ] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Datafeed started (from: 1970-01-21T11:17:02.401Z to: 2026-01-28T03:36:40.546Z) with frequency [450000ms]
[2026-01-28T16:36:43,769][WARN ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] Datafeed lookback retrieved no data
[2026-01-28T16:36:43,769][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] attempt to stop datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories]
[2026-01-28T16:36:43,769][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] try lock [20s] to stop datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories]...
[2026-01-28T16:36:43,769][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] stopping datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories], acquired [true]...
[2026-01-28T16:36:43,769][INFO ][o.e.x.m.d.DatafeedRunner ] [Ed’s-MacBook-Pro-M4] [no_realtime] datafeed [datafeed-logs_services_count_logs_categories] for job [logs_services_count_logs_categories] has been stopped
[2026-01-28T16:36:43,930][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [Ed’s-MacBook-Pro-M4] Closing job [logs_services_count_logs_categories], because [close job (api)]
[2026-01-28T16:36:43,931][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] [autodetect/20172] [CCmdSkeleton.cc@66] Handled 0 records
[2026-01-28T16:36:43,931][INFO ][o.e.x.m.p.AbstractNativeProcess] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] State output finished
[2026-01-28T16:36:43,931][INFO ][o.e.x.m.j.p.a.o.AutodetectResultProcessor] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] 0 buckets parsed from autodetect output
[2026-01-28T16:36:43,933][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [Ed’s-MacBook-Pro-M4] [logs_services_count_logs_categories] autodetect connection for job closed

[valeriy42]

@edsavage, this is great progress. 🚀 Thank you very much! From the error message, it seems the failure is recoverable, since we can re-categorize from scratch, right? (Sorry, I don't have the whole algorithm in my head.)

If this is recoverable, please downgrade the messages from ERROR to WARNING, as ERROR will still create unnecessary noise when we monitor the services.