Mastodon Collector

The Mastodon Collector imports posts from a Mastodon instance through scheduled API polling. It supports hashtag timelines, the access-token owner’s home timeline, and public posts from a specific account.

Use the native collector when you need authenticated access, reliable pagination, or collection progress across runs. For simple public feeds, the RSS alternative may be sufficient.

Timeline modes

Mastodon instances decide whether anonymous hashtag access is allowed. A hashtag can therefore work without a token on one instance and require authentication on another. The instance’s federation state, moderation policy, and knowledge of remote accounts also determine which posts are available; an instance is not a complete index of the Fediverse.

See Mastodon’s API documentation for the hashtag and home timeline endpoints, account lookup and account-status endpoints, and OAuth scopes.

Create an OSINT source

1. Open Administration → OSINT Source.
2. Select New OSINT Source.
3. Enter a name and select Mastodon Collector.
4. Configure the parameters described below.
5. Save the source.
6. Use Preview to verify the result without adding posts to Assess.
7. Use Collect when the preview is correct, or enable the source for scheduled collection.

Parameters

The global Collector Entry Limit in Administration → Settings limits the number of posts processed during each run.

Example: #curl on chaos.social

If the token field is empty and the instance restricts anonymous hashtag access, collection fails with the actionable access-token-required message described under Troubleshooting.

Create a Mastodon access token

Mastodon’s web interface places access-token management under Development, not under the normal account or privacy settings. The token must be created on the instance configured in INSTANCE_URL; a token from another instance will not work.

1. Sign in to the Mastodon instance used by the source.
2. Open Preferences → Development. The direct URL is normally https://INSTANCE/settings/applications. For example, use chaos.social Development for https://chaos.social.
3. Select New application.
4. Enter an application name such as Taranis AI.
5. Leave Application website empty unless your organization wants to identify the deployment.
6. Leave the default redirect URI urn:ietf:wg:oauth:2.0:oob. Taranis does not use an interactive OAuth redirect.
7. Select only the scopes needed for the configured timeline:
   • Hashtag: read:statuses.
   • Home: profile and read:statuses.
   • Account: read:accounts and read:statuses.
8. Submit the application.
9. Open the created application and copy Your access token. Do not copy the client secret instead.
10. Paste the token into the Taranis source’s ACCESS_TOKEN field and save the source.

Some Mastodon versions preselect profile on the new-application form. Clear it for hashtag or account collection when it is not listed above. For one application used by all three modes, select profile, read:accounts, and read:statuses. Do not select write, follow, push, or any admin scope. On Mastodon versions older than 4.3, use read:accounts instead of profile for home-timeline collection.

Mastodon’s developer documentation describes user tokens, granular scopes, and the full OAuth authorization flow. The Development page is the shortest setup path for a Taranis administrator who controls the Mastodon account.

Token security

• Treat the access token like a password. Do not paste it into tickets, chat, screenshots, shell history, or logs.
• Prefer a dedicated organizational Mastodon account rather than a personal account for production collection.
• Grant only the scopes listed for the selected timeline. The collector never needs write or administrator access.
• Taranis masks the stored secret after saving. Revealing or replacing it is an audited administrator action.
• If a token may have leaked, delete or revoke the Mastodon application, create a replacement, and update the Taranis source.

Collection behavior

The collector polls Mastodon’s REST API; it does not open a long-lived streaming connection and does not update Mastodon timeline markers.

On the first successful run, Taranis imports the newest posts up to the global Collector Entry Limit. Later runs request posts newer than the saved cursor. The cursor is stored with the latest collector task result and advances only after the collected News Items are published successfully. A failed Mastodon request or failed publication does not advance it.

Cursor state is intentionally best-effort. Deleting task history, retaining no task result for an inactive source, or restoring a database without recent task results can remove it. The next run then starts again from the newest posts. Core deduplication makes replay safe, but posts older than the entry limit may be skipped after such a reset.

Changing the instance, timeline mode, hashtag, token owner, or target account establishes a new timeline identity and starts a new cursor. Preview ignores the cursor and never changes collection progress.

Mapping and deduplication

• Each Mastodon post becomes a News Item.
• Boosts are stored as the original post and deduplicate through the original post URL.
• Replies and boosts visible in the selected timeline are retained.
• A content warning becomes the title when present; otherwise Taranis derives the title from the post content.
• Media descriptions provide content when a post has no text.
• The source account, author, publication time, language, post URL, and plain-text content are retained when supplied by Mastodon.

See Mastodon’s Status entity for the upstream response fields.

Troubleshooting

Access token required

Message: This Mastodon instance requires an access token to collect hashtags

The instance does not permit the combined hashtag feed to be read anonymously. Create a token with read:statuses, add it to the source, and preview again. This is expected on instances that expose local hashtag posts publicly but require authentication for remote posts.

Mastodon documents this behavior on the hashtag timeline endpoint.

Authentication failed or access was denied

Verify that:

• The token was created on exactly the same instance as INSTANCE_URL.
• The token was copied from Your access token, not Client secret.
• The application has the scopes required for the selected mode.
• The application or token has not been revoked.
• The Mastodon account is still active and permitted to use the timeline.

Timeline or account not found

• Enter a hashtag without a URL. A leading # is optional.
• Enter an account as user@example.social, not as a profile URL.
• Confirm that the account is visible to and known by the instance in INSTANCE_URL.
• Open the hashtag or account from the instance’s own web interface to confirm that it exists there.

Rate limit exceeded

Increase REFRESH_INTERVAL or the global collector interval. Mastodon applies rate limits per account and IP address; see the Mastodon rate-limit documentation.

Instance unavailable

Check DNS, TLS certificates, firewall rules, the configured proxy, and whether the instance API is reachable from the collector worker. INSTANCE_URL must be the instance origin, not a profile, hashtag, or API URL.

No new statuses

This is a normal not-modified result. It means the timeline contains no posts newer than the saved cursor, or all returned posts already exist in Taranis.

Official Mastodon references

• Timeline API: hashtag and home timelines
• Accounts API: credential verification, lookup, and statuses
• OAuth token types
• OAuth scopes and least-privilege guidance
• OAuth authorization flow
• Status response fields
• API pagination
• API rate limits