Sign In

Self-Hosted Troubleshooting

Please keep in mind that the self-hosted repository is geared towards low to medium loads with simplicity in mind. Folks needing larger setups or having event spikes can expand from here based on their specific needs and environments. If this is not your cup of tea, you are always welcome to try out hosted Sentry.

General

You can see the logs of each service by running docker-compose logs <service_name>. You can use the -f flag to "follow" the logs as they come in, and use the -t flag for timestamps. If you don't pass any service names, you will get the logs for all running services. See the reference for the logs command for more info.

Kafka

One of the most likely things to cause issues is Kafka. The most commonly reported error is

Copied
Exception: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}

This happens where Kafka and the consumers get out of sync. Possible reasons are:

  1. Running out of disk space or memory
  2. Having a sustained event spike that causes very long processing times, causing Kafka to drop messages as they go past the retention time
  3. Date/time out of sync issues due to a restart or suspend/resume cycle

Recovery

Note: These solutions may result in data loss when resetting the offset of the snuba consumers.

Proper solution

The proper solution is as follows (reported by @rmisyurev):

  1. Receive consumers list:
    Copied
    docker-compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --list
  2. Get group info:
    Copied
    docker-compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers -describe
  3. Watching what is going to happen with offset by using dry-run (optional):
    Copied
    docker-compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --topic events --reset-offsets --to-latest --dry-run
  4. Set offset to latest and execute:
    Copied
    docker-compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --topic events --reset-offsets --to-latest --execute

Another option

This option is as follows (reported by @gabn88):

  1. Set offset to latest and execute:
    Copied
    docker-compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --all-groups --all-topics --reset-offsets --to-latest --execute

Unlike the proper solution, this involves resetting the offsets of all consumer groups and all topics.

Nuclear option

The nuclear option is removing all Kafka-related volumes and recreating them which will cause data loss. Any data that was pending there will be gone upon deleting these volumes.

  1. Stop the instance:

    Copied
    docker-compose down --volumes

  2. Remove the Kafka & Zookeeper related volumes:

    Copied
    docker volume rm sentry-kafka
docker volume rm sentry-zookeeper

  3. Run the install script again:

    Copied
    ./install.sh

  4. Start the instance:

    Copied
    docker-compose up -d

Reducing disk usage

If you want to reduce the disk space used by Kafka, you'll need to carefully calculate how much data you are ingesting, how much data loss you can tolerate and then follow the recommendations on this awesome StackOverflow post or this post on our community forum.

Redis

Redis is used both as a transactional data store and a work queue of Celery in the self-hosted setup. For this reason, it may get overwhelmed during event spikes. We have made some significant improvements regarding this starting from version 20.10.1. If you are still having issues, you may look into scaling out Redis itself or switching to a different Celery broker, such as RabbitMQ.

Workers

If you are seeing an error such as

Copied
Background workers haven’t checked in recently. It seems that you have a backlog of 200 tasks. Either your workers aren’t running or you need more capacity.

you may benefit from using additional, dedicated workers. This is achieved by creating new worker services in docker-compose.override.yml and tying them to specific queues using the -Q queue_name argument. An example would be:

Copied
worker1:
    << : *sentry_defaults
    command: run worker -Q events.process_event

To see a more complete example, please see a sample solution on our community forum.

Postgres

Postgres is used for the primary datastore, as well as the nodestore which is used to store key/value data. The nodestore_node table can grow rapidly, especially when heavily utilising the Performance Monitoring feature as trace data is stored in this table.

The nodestore_node table is cleaned up as part of the cleanup task, however Postgres may not get a chance to vacuum the table (especially under heavy load), so even the rows may be deleted, they're still taking up space on disk.

You can use pg-repack which repacks a table live by creating a new table and copying data across, before dropping the old one. You'll want to run this after the clean up script, and note that as it creates a table, disk usage will spike before going back down.

An example script below:

Copied
# Only keep the last 7 days of nodestore data. We heavily use performance monitoring.
docker-compose run -T web cleanup --days 7 -m nodestore -l debug
# This ensures pg-repack exists before running as the container gets recreated on upgrades
docker-compose run -T postgres bash -c "apt update && apt install -y --no-install-recommends postgresql-9.6-repack && su postgres -c 'pg_repack -E info -t nodestore_node'"

If the script above still does not clear up enough disk space, you can try the following in Postgres. This will lead to data loss in event details. SENTRY_RETENTION_DAYS will need to be filled in manually.

Copied
DELETE FROM public.nodestore_node WHERE “timestamp” < NOW() - INTERVAL ‘[SENTRY_RETENTION_DAYS]’;
VACUUM FULL public.nodestore_node;

VACUUM FULL actively compacts tables by writing a complete new version of the table file with no dead space. This minimizes the size of the table, but can take a long time. It also requires extra disk space for the new copy of the table, until the operation completes.

Docker Containers' Healthcheck

There may be some circumstances which you may want to increase or decrease healthcheck interval, timeout or retries for your custom needs. This can be achieved by editing HEALTHCHECK_INTERVAL, HEALTHCHECK_TIMEOUT, HEALTHCHECK_RETRIES variables' values in .env.

Other

If you are still stuck, you can always visit our GitHub issues to search for existing issues or create a new issue and ask for help. Please keep in mind that we expect the community to help itself, but Sentry employees also try to monitor and answer questions when they have time.

You can edit this page on GitHub.