Skip to main content

Edge Agent Troubleshooting

Most problems fall into one of a few buckets: the agent can't reach Haltless, readings are being rejected, or a source device isn't responding. Start with the logs: they name the cause in almost every case.

Reading the logs​

When the agent runs under systemd, its logs go to the journal:

# Follow live
sudo journalctl -u haltless-agent -f

# Last 200 lines
sudo journalctl -u haltless-agent -n 200

# Since a point in time
sudo journalctl -u haltless-agent --since "1 hour ago"

If you run the agent in the foreground, the same messages print to the terminal. For more detail during diagnosis, set log_level: "DEBUG" in config.yaml and restart.

The key line to look for is the heartbeat: it reports the agent's connection status (active, or a reason it can't send) on a regular interval.

Connectivity and firewalls​

Symptom: log messages about connection errors or timeouts; the agent reports it is buffering locally.

The agent needs only outbound HTTPS to api.haltless.io on port 443. Nothing needs to connect into the agent.

  1. Confirm the host can reach Haltless. A 2xx here means the path is open: 
    curl -sS -o /dev/null -w "%{http_code}\n" https://api.haltless.io/health
  2. Check DNS resolves: nslookup api.haltless.io.
  3. Confirm your firewall allows outbound 443 to that host.
  4. If you route through a corporate proxy, make sure the agent's environment is configured to use it (for example, an HTTPS_PROXY value in the service environment).

While Haltless is unreachable, the agent keeps collecting and writes readings to its local buffer. It automatically resumes sending, oldest data first, once the connection is restored, so a brief outage is invisible downstream.

The first send that finds no connection is saved to the buffer straight away; the agent does not stop collecting to retry it. It checks the connection again on the next cycle, then 30 seconds later, then less often while the outage lasts (at most every 5 minutes), so sending resumes within that interval of Haltless coming back. In the log you'll see send_batch no connection and buffered N readings locally when the connection drops, and Flushing N buffered batches once it returns. Buffered readings go out in requests of up to 1,000 readings (Flushed N buffered readings), so even a long outage takes few requests to catch up.

Buffering during outages​

The agent is designed to ride out network interruptions. When it can't send, it persists readings to a crash-safe on-disk buffer (buffer.db in buffer_dir) and drains them in order when connectivity returns.

Two limits keep the buffer bounded, and both mean data loss only in a sustained outage:

  • Size cap (max_buffer_size_mb, default 512): once the buffer reaches this size while offline, the newest readings are dropped.
  • Retention (~24 hours): buffered readings older than about a day are evicted automatically.

If you expect long outages, raise max_buffer_size_mb and make sure buffer_dir has enough free disk.

Disk space for the buffer​

Symptom: a growing buffer.db, or the agent warns it is dropping readings because the buffer is full.

A growing buffer almost always means the agent can't reach Haltless. Fix connectivity first (see above) and the buffer will drain on its own.

  • Check the buffer size: ls -lh /var/lib/haltless-agent/buffer.db (adjust for your buffer_dir).
  • Make sure buffer_dir is on a volume with enough free space for your worst-case offline window.
  • Prefer fast local storage (an SSD) for buffer_dir so draining a large backlog is quick.
  • Keep buffer_dir on a normal data path such as /var/lib/haltless-agent. System directories are rejected at startup.

API-key and authentication errors​

Symptom: the log reports API key refused (status=401) or (status=403) and says it is holding readings on disk, or the heartbeat reports heartbeat rejected.

A refused key is not the readings' fault, so the agent does not drop them. When Haltless refuses the key (401: wrong, expired or revoked; 403: it lacks the ingest scope), the agent stops sending and keeps collecting into its local buffer, as it does during an outage. It asks again at every heartbeat (every 5 minutes) and, once the key is accepted, sends the backlog oldest first. The readings wait within the buffer's limits (see Buffering during outages), so fix the key within about a day to lose nothing. To fix it:

  1. Confirm the api_key in config.yaml is correct and has not been revoked.
  2. Check for stray whitespace or a truncated value. The key must be pasted in full.
  3. For a 403, give the key the ingest scope. An edge-agent key created without scopes has it. See API keys.
  4. If in doubt, create a fresh key and update the config.
  5. Restart the agent after changing the key in config.yaml. What it buffered meanwhile is sent with the new key. A fix made in Haltless instead (a scope added, a key re-enabled) needs no restart: it takes effect at the next heartbeat.

A 403 on the heartbeat alone is different: heartbeat refused (403): the API key lacks the 'agent' scope means readings keep flowing, but Haltless does not record the agent's check-ins or version until the key gets the agent scope.

Collection paused (company suspended)​

Symptom: the heartbeat reports a payment-required (402) status and the agent pauses collection.

This means your company's Haltless account has been suspended. The agent intentionally stops collecting so it doesn't pile up data that can't be delivered. Contact your Haltless administrator or Haltless support to have the account reinstated; the agent keeps checking in (about once an hour while paused) and automatically resumes, flushing anything it had buffered, once the account is active again. Nothing needs to change in the agent's configuration.

Rejected readings​

If you see rejections, check the reason. Common ones are below. See the API error reference for the full list.

System identifier not found. The reading's System identifier doesn't match an active System in your company. Haltless rejects just that reading and stores the rest of the batch. Identifiers are case-sensitive; LINE-3 and line-3 are different Systems. Create the System on the portal's Systems page, or correct system_identifier in the source.

The next three make Haltless refuse the whole request (422). The agent then splits the batch to find the offending readings, drops and logs them, and sends the rest.

Value must be finite. The value was NaN or infinite. For Modbus, check the scale and offset: a bad conversion can produce a non-finite result. Also check the source device for fault states that emit non-numeric values.

Missing unit. Every reading needs a unit. Make sure your register/node config sets unit, or that your CSV column / JSON key for the unit is populated.

Timestamp out of range. The reading's timestamp is more than 5 minutes in the future or more than 3,650 days in the past. This is almost always a clock problem (see below).

Readings arrive but nothing is monitored. A tag the System has not sent before waits, unassigned, until someone assigns it to a machine on the System's page. Its readings are stored meanwhile and come along when it is assigned.

Timestamp and clock issues​

Symptom: readings rejected for an out-of-range timestamp, or data appearing at the wrong time on the dashboard.

  • Timestamps are handled in UTC. For the PLCs and controllers (Modbus TCP and RTU, OPC-UA, S7, EtherNet/IP, MELSEC, FINS, ADS, and BACnet/IP), the agent stamps readings with the host's current time, so keep the host clock synchronized with NTP.
  • For MQTT, a message's own timestamp is used when it has one, so the publishing device's clock matters too. A message time more than two minutes ahead of the agent host's clock is replaced by the arrival time and logged once per tag. Sparkplug B works the same way with each metric's own timestamp.
  • For PI Web API, a value keeps the time PI recorded it. One ahead of the agent host's clock takes the agent's time, so check the PI server's clock too.
  • For CSV and JSON, the timestamp comes from your export. If it has no timezone offset, the agent assumes UTC and logs a one-time warning, so make sure your exports really are UTC, or include an explicit offset.
  • If readings land in the future or far in the past, check both the source device clock and the agent host clock.

The agent replays buffered readings on reconnect; re-sending the same reading is safe and does not create duplicates.

Rate limiting​

Symptom: the log reports that a batch was rate-limited and buffered for retry.

Ingest requests are limited to 600 per minute from each network address, so every agent behind the same plant address shares that budget. If you send faster than that, Haltless returns a rate-limit response. The agent buffers the batch and retries it later rather than dropping it. To reduce pressure, increase collection_interval_seconds, raise batch_size (up to 1000) so you make fewer, larger requests, or reduce the number of high-frequency sources. See Rate limiting for the limits.

Source device won't connect​

Modbus: no response.

  1. Confirm network reachability to the device on its port (default 502).
  2. Verify the host and port in the config.
  3. Confirm the device speaks Modbus TCP. A device on a serial line needs a modbus_rtu source instead, or a gateway with unit_id set.
  4. If values look wrong, re-check address, register_type (holding vs input), and scale/offset. Remember count: 2 reads a big-endian 32-bit float.

OPC-UA: connection or node errors.

  1. Verify the endpoint_url uses the opc.tcp:// scheme and the host/port are reachable.
  2. If the server needs credentials, set both username and password; setting only one causes an anonymous connection.
  3. If a specific node fails, confirm the node_id exists on the server using an OPC-UA browser tool.

Siemens S7: connection refused or every value fails.

  1. Confirm the PLC is reachable on port 102 from the agent host.
  2. Check rack and slot. A wrong slot shows only as a refused connection: an S7-300 is usually slot 2, an S7-1200 or S7-1500 slot 1, and an S7-400's is in its hardware configuration.
  3. On an S7-1200 or S7-1500, tick Permit access with PUT/GET communication from remote partner in the CPU's protection settings, and switch Optimized block access off on each data block you read. Both need a download to the CPU.
  4. If one value fails with "Object does not exist" or "Invalid address", the data block is missing or shorter than the address. Check the offset against the block in TIA Portal or STEP 7.
  5. If the agent will not start, the log names the machine, the tag's position, and its address. The usual cause is a size letter that does not match the type, for example DBW with a REAL.

EtherNet/IP: connection or tag errors.

  1. Confirm the controller is reachable on port 44818 from the agent host.
  2. For a ControlLogix, set slot to the controller's slot in the chassis. A CompactLogix or Micro800 needs 0.
  3. A tag that does not exist is skipped and logged. Program-scoped tags need their Program:<name>. prefix, exactly as in Studio 5000.
  4. A tag that holds a STRING, a whole UDT, or a whole array is skipped: point at a numeric member or element instead.
  5. For an SLC 500 or MicroLogix, set controller: slc and use data-file addresses such as N7:0 or F8:2.

Modbus RTU: no response or garbled values.

  1. Confirm port names the serial adapter (ls /dev/ttyUSB*). In Docker, check the device is passed into the container and the agent's user has its group.
  2. Match baudrate, parity, and stopbits to the devices. Many ship at 9600 baud with no parity (parity: N).
  3. Check each register's unit_id against the device's address. On RS-485, check the A/B wiring and the line termination.
  4. A second source on the same port is refused at startup: list every device on the bus under one source.

Mitsubishi MELSEC: connection refused or every value fails.

  1. Confirm the CPU has an MC protocol (SLMP) connection open on port, set to TCP and binary code. A connection set to ASCII code or UDP does not answer the agent.
  2. On an iQ-F (FX5), set xy_numbering: octal.
  3. A device logged as refused, with an end code, does not exist on the CPU or is out of range. Check its number against the CPU's device settings.

Omron FINS: no answer, or addresses refused.

  1. Confirm the PLC is reachable on UDP port 9600 from the agent host.
  2. If the PLC's FINS settings use an IP address table instead of automatic address generation, set destination_node and source_node to match.
  3. An address logged as refused, with an end code, is outside the CPU's memory. Check the area and word number.

Beckhoff ADS: connection or symbol errors.

  1. Add a route on the PLC (TwinCAT, System, Routes) for the agent's IP address and AMS Net ID. Without one the PLC does not answer.
  2. Check ams_port: 851 for TwinCAT 3's first PLC, 801 for TwinCAT 2.
  3. A symbol logged as refused is not in the PLC program under that name. Check the full path, such as MAIN.speed or GVL.Pumps[2].count.
  4. One cycle reports an error after a download or online change, while the agent reconnects for the new symbol table. That is expected.
  5. Every ADS source on one agent must use the same local_ams_net_id. A source with a different one is refused at startup.

BACnet/IP: no answer, or objects refused.

  1. Confirm the device is reachable on UDP port 47808 and that the firewall lets its answers back to the agent. If the firewall needs a fixed port for the agent, set local_port.
  2. A device behind a BACnet router (an MS/TP field bus) is not reached yet. Read a device with its own IP address.
  3. An object logged as refused does not exist on the device, or does not have that property. Check the type and instance in the device's object list.

MQTT: not connected, or no readings.

  1. Confirm the broker is reachable on its port (1883, or 8883 over TLS) from the agent host. The log gives the reason the first connection failed (refused, a name that does not resolve, a TLS error such as TLS against a plain listener), the broker's reason when it refuses the login or a subscription or drops the connection, and one warning while it stays unreachable.
  2. Check that each filter in topics matches what your plant publishes. Running mosquitto_sub -h <broker> -t 'plant/line3/#' -v on the agent host shows the topics and payloads the agent will see.
  3. A topic logged as having no number to read publishes text, an array, or a payload that is not UTF-8. Point topics at the numeric topics, or publish a JSON object with numeric fields.
  4. Two clients with the same client_id knock each other off the broker. Give each agent its own, or leave the default.
  5. A tag that stops reporting has had no new message for hold_seconds. Raise hold_seconds for topics that publish rarely.

Sparkplug B: no readings.

  1. Check the broker connection as for MQTT, then group_id, edge_node_id, and device_id against what the edge nodes publish: mosquitto_sub -h <broker> -t 'spBv1.0/Plant1/#' -v shows their topics.
  2. Values sent by alias are skipped until the agent has seen the node's birth. The agent sends no rebirth requests, so restart the edge node or have it publish a new birth.
  3. Strings, data sets, templates, and historical (store-and-forward) metrics are not read.

AVEVA PI Web API: refused, or no values.

  1. Confirm base_url opens from the agent host, and that Basic authentication is enabled on the PI Web API server.
  2. A refused login (401 or 403) means the username or password is wrong, or the account lacks read access in PI.
  3. A path logged as not found does not exist under that name. Check it in PI System Explorer, and write it in single quotes in config.yaml so YAML keeps its backslashes.
  4. A point in a system state such as Shutdown or I/O Timeout is skipped until it reads a value again.

CSV / JSON: no readings.

  1. Confirm the watched directory exists and the agent's user can read it.
  2. Check that column names (col_*) or object keys (key_*) match your files exactly.
  3. Remember CSV files are read incrementally (only new rows each cycle) and JSON/JSONL files are processed once, so drop each new export in as a new file.

Still stuck?​

Collect the recent logs with the API key and any passwords redacted, note the agent version and the exact error text, and share them with your Haltless support contact. The Getting started guides and the API reference cover the wider platform if the issue turns out to be downstream of the agent.