Troubleshooting

​

Quick Diagnostics

# System compatibility check
oisp-sensor check

# SSL library detection
oisp-sensor ssl-info

# Service status (if running as service)
sudo systemctl status oisp-sensor

# View recent logs
sudo journalctl -u oisp-sensor -n 100

​

Installation Issues

​

”Permission denied” when running

1. Run with sudo (simplest):
   Copy

   sudo oisp-sensor
   
2. Set capabilities (run without sudo):
   Copy

   sudo setcap cap_sys_admin,cap_bpf,cap_perfmon,cap_net_admin+ep /usr/bin/oisp-sensor
   oisp-sensor  # Now works without sudo
   
3. Verify capabilities:
   Copy

   getcap /usr/bin/oisp-sensor
   # Should show: cap_sys_admin,cap_bpf,cap_perfmon,cap_net_admin=ep
   

​

“BTF not found”

ls /sys/kernel/btf/vmlinux
# If file doesn't exist, BTF is not available

# Install kernel headers
sudo apt-get install linux-headers-$(uname -r)

# Or upgrade to a kernel with BTF (5.8+)
sudo apt-get update
sudo apt-get upgrade linux-image-generic
sudo reboot

# Install kernel-devel
sudo dnf install kernel-devel

# Or upgrade kernel
sudo dnf update kernel
sudo reboot

uname -r  # Check kernel version (should be 5.8+)
ls /sys/kernel/btf/vmlinux  # Should exist

​

“No SSL libraries found”

ldconfig -p | grep libssl

sudo apt-get install libssl3
# Or for older systems:
sudo apt-get install libssl1.1

sudo dnf install openssl-libs

oisp-sensor ssl-info
# Should show found libraries

​

“Failed to attach eBPF program”

1. Kernel too old (< 4.18)
2. Missing kernel config options
3. Resource limits too low

1. Check kernel version:
   Copy

   uname -r  # Should be >= 4.18, preferably >= 5.8
   
2. Check dmesg for errors:
   Copy

   sudo dmesg | grep -i bpf
   
3. Increase resource limits:
   Copy

   ulimit -l unlimited
   
4. Or via systemd:
   Copy

   sudo systemctl edit oisp-sensor
   
   Add:
   Copy

   [Service]
   LimitMEMLOCK=infinity
   
5. Verify BPF filesystem:
   Copy

   ls /sys/fs/bpf  # Should exist
   

​

Package installation fails

sudo apt-get install -f
# This fixes missing dependencies

# Install without GPG check (not recommended for production)
sudo dnf install --nogpgcheck ./oisp-sensor-0.2.0-1.x86_64.rpm

​

Runtime Issues

​

No events appearing

1. Check if sensor is actually running:
   Copy

   ps aux | grep oisp-sensor
   
2. Check if eBPF programs are loaded:
   Copy

   sudo bpftool prog list | grep oisp
   # Should show uprobe programs
   
3. Check logs for errors:
   Copy

   sudo journalctl -u oisp-sensor -n 50
   
4. Verify AI activity is happening:
   • Are you making actual API calls?
   • Is your API key set? (echo $OPENAI_API_KEY)
5. Check if SSL capture is working:
   Copy

   # Enable debug logging
   RUST_LOG=debug sudo oisp-sensor record --output /tmp/test.jsonl
   
   # In another terminal, make HTTPS call
   curl -v https://api.openai.com/v1/models
   
6. Check SSL library detection:
   Copy

   oisp-sensor ssl-info
   # Should show detected libraries
   

• NVM Node.js or pyenv Python - See Edge Cases below
• Go applications - Use different TLS implementation (not yet supported)
• Applications using rustls - Rust-native TLS (not yet supported)

​

Events captured but incomplete

cat /etc/oisp/config.toml | grep mode

1. Enable full capture (for debugging):
   Copy

   sudo oisp-sensor record --redaction-mode full --output /tmp/test.jsonl
   
2. Or update config:
   Copy

   [redaction]
   mode = "full"  # safe, full, or minimal
   

​

High CPU usage

# Check CPU usage
top -p $(pgrep oisp-sensor)

# Check event rate
sudo journalctl -u oisp-sensor -f | grep "Captured"

• Very high event rate (>1000 events/sec)
• Too many processes being monitored
• TUI/Web UI enabled in high-load environment

1. Filter by specific processes:
   Copy

   sudo oisp-sensor --comm python3,node
   
2. Disable UI in production:
   Copy

   sudo oisp-sensor record --no-web --output /var/log/oisp/events.jsonl
   
3. Increase systemd CPU limit:
   Copy

   sudo systemctl edit oisp-sensor
   
   Add:
   Copy

   [Service]
   CPUQuota=200%  # Limit to 2 cores
   

​

High memory usage

ps aux | grep oisp-sensor

• Very large events (huge prompts/responses)
• Memory leak (rare, please report!)
• Buffer not draining to export

1. Check export is working:
   Copy

   ls -lh /var/log/oisp/events.jsonl
   # File should be growing
   
2. Enable batched export:
   Copy

   [export.jsonl]
   batch_size = 100
   
3. Set memory limit:
   Copy

   sudo systemctl edit oisp-sensor
   
   Add:
   Copy

   [Service]
   MemoryMax=512M
   

​

Service keeps restarting

# Check restart count
sudo systemctl status oisp-sensor

# Check recent logs
sudo journalctl -u oisp-sensor --since "1 hour ago"

1. Out of memory:
   Copy

   sudo journalctl -u oisp-sensor | grep -i "killed"
   
   Solution: Increase memory or set limits
2. Permission issues:
   Copy

   sudo journalctl -u oisp-sensor | grep -i "permission denied"
   
   Solution: Check file permissions, capabilities
3. Config file errors:
   Copy

   # Test config manually
   sudo oisp-sensor record --config /etc/oisp/config.toml --output /tmp/test.jsonl
   

​

Edge Cases: NVM/pyenv

​

NVM Node.js not captured

# Find your Node.js binary
which node
# Example: /home/user/.nvm/versions/node/v20.10.0/bin/node

# Run sensor with binary path
sudo oisp-sensor --binary-path /home/user/.nvm/versions/node/v20.10.0/bin/node

# /etc/oisp/config.toml
[capture.ssl]
binary_paths = [
    "~/.nvm/versions/node/*/bin/node",
]

# Check if sensor attaches to NVM Node
sudo journalctl -u oisp-sensor | grep "Attached uprobe"
# Should show: Attached uprobe to /home/user/.nvm/.../node

​

pyenv Python not captured

# /etc/oisp/config.toml
[capture.ssl]
binary_paths = [
    "~/.pyenv/versions/*/bin/python*",
]

sudo oisp-sensor --binary-path ~/.pyenv/versions/3.11.0/bin/python

​

Docker containers not captured

​

Export Issues

​

Events not writing to file

# Check if file exists
ls -lh /var/log/oisp/events.jsonl

# Check directory permissions
ls -ld /var/log/oisp/

# Check sensor is writing
sudo lsof | grep events.jsonl

1. Create directory:
   Copy

   sudo mkdir -p /var/log/oisp
   sudo chown oisp:oisp /var/log/oisp
   
2. Check permissions:
   Copy

   sudo chmod 755 /var/log/oisp
   
3. Verify config:
   Copy

   [export.jsonl]
   enabled = true
   path = "/var/log/oisp/events.jsonl"
   

​

OTLP export failing

# Check network connectivity
curl -v http://collector:4317

# Check sensor logs
sudo journalctl -u oisp-sensor | grep -i otlp

1. Verify endpoint:
   Copy

   [export.otlp]
   enabled = true
   endpoint = "http://collector:4317"  # Not https!
   
2. Check firewall:
   Copy

   # Test connection
   telnet collector 4317
   
3. Enable debug logging:
   Copy

   RUST_LOG=debug sudo oisp-sensor record
   

​

Debug Mode

​

Via command line

RUST_LOG=debug sudo oisp-sensor record --output /tmp/debug.jsonl

​

Via systemd

sudo systemctl edit oisp-sensor

[Service]
Environment="RUST_LOG=debug"

sudo systemctl daemon-reload
sudo systemctl restart oisp-sensor
sudo journalctl -u oisp-sensor -f

​

Log levels

• RUST_LOG=error - Errors only
• RUST_LOG=warn - Warnings and errors
• RUST_LOG=info - Default level
• RUST_LOG=debug - Verbose debugging
• RUST_LOG=trace - Very verbose (includes eBPF events)

​

Module-specific logging

# Only eBPF module debug logs
RUST_LOG=oisp_capture_ebpf=debug sudo oisp-sensor

# Multiple modules
RUST_LOG=oisp_capture_ebpf=debug,oisp_decode=debug sudo oisp-sensor

​

Getting Help

​

Collect Diagnostics

1. System check:
   Copy

   oisp-sensor check > system-check.txt
   
2. SSL info:
   Copy

   oisp-sensor ssl-info > ssl-info.txt
   
3. Service logs:
   Copy

   sudo journalctl -u oisp-sensor --since "1 hour ago" > sensor-logs.txt
   
4. System info:
   Copy

   uname -a > system-info.txt
   cat /etc/os-release >> system-info.txt
   
5. eBPF programs:
   Copy

   sudo bpftool prog list > bpf-programs.txt
   

​

Report an Issue

• Output from diagnostic commands above
• Steps to reproduce
• Expected vs actual behavior
• Anonymized config file (remove secrets!)

​

Community Support

• GitHub Discussions: https://github.com/oximyhq/sensor/discussions
• Documentation: https://sensor.oximy.com

​

Next Steps

• Production Deployment - Optimize for production
• Running as a Service - Manage systemd service
• Distribution Support - Check compatibility