This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Collectors

Collectors are used to gather data from various sources.

Supported options

  1. RSS Collector
  2. Simple Web Collector
  3. RT Collector
  4. MISP Collector
  5. Mastodon Collector

The administration view now allows users to use the Preview feature to see the result of the configuration without the items being processed further for the Assess view. This feature is available for RSS, Simple Web, RT, and Mastodon collectors.

RSS Collector

RSS Collector enables Taranis AI to collect data from a user-defined RSS feed (See RSS feeds details).

  • Required fields:
    • FEED_URL
  • Optional fields:
    • USER_AGENT
    • PROXY_SERVER
    • USE_GLOBAL_PROXY (ignore anything set in the PROXY_SERVER field, use what is currently set as a default in Settings; learn more in section Settings)
    • ADDITIONAL_HEADERS [accepts a valid json] (can be used to add additional headers, not all headers work as expected)
    • CONTENT_LOCATION (selects which field from the RSS entry should be used as the item content, for example description or content:encoded; when the collector uses content from the RSS entry, it first checks the field named here; if that field is empty or missing, it falls back through content, content:encoded, summary, and description)
    • USE_FEED_CONTENT (controls whether the collector uses content from the RSS entry itself or fetches the linked article page; if this is enabled, the collector uses RSS entry content; if it is disabled, the collector fetches the linked article page; if it is not set, a non-empty CONTENT_LOCATION effectively enables using RSS entry content)
    • XPATH (optional XPath expression used to limit the extracted content to a specific element; if USE_FEED_CONTENT is enabled, XPath is applied to the HTML content taken from the RSS entry; if the XPath does not match, the collector keeps the original RSS entry content; if USE_FEED_CONTENT is disabled, XPath is applied to the fetched article page; if the XPath does not match there, the collector does not fall back to another extraction method and the content may be empty)
    • TLP_LEVEL
    • REFRESH_INTERVAL (see Bots - refresh_interval)
    • DIGEST_SPLITTING On/Off (creates News Items out of URLs present in the Summary field of RSS feed)
    • DIGEST_SPLITTING_LIMIT (default: 30)
    • BROWSER_MODE On/Off (see Browser Mode)

Basic configuration

Advanced configuration

The RSS Collector supports the use of XPath for locating elements. (See Simple Web Collector Advanced configuration)

  • Example:
    • FEED_URL: https://www.bmi.gv.at/rss/bmi_presse.xml
    • XPATH: //*[@id=“getting-started-web-console-creating-new-project_openshift-web-console”]
    • ADDITIONAL_HEADERS: { "AUTHORIZATION": "Bearer Token1234", "X-API-KEY": "12345", "Cookie": "firstcookie=1234; second-cookie=4321", }

Simple Web Collector

Simple Web Collector enables Taranis AI to collect data using web URLs and XPaths.

  • Required field:
    • WEB_URL
  • Optional fields:
    • USER_AGENT
    • PROXY_SERVER
    • USE_GLOBAL_PROXY
    • ADDITIONAL_HEADERS
    • XPATH (set to specify the location of the scraped element on the website)
    • TLP_LEVEL
    • DIGEST_SPLITTING On/Off
    • DIGEST_SPLITTING_LIMIT (default: 30)
    • BROWSER_MODE On/Off (see Browser Mode)

Basic configuration

The simplest way to use this collector is to use the WEB_URL field only. By using only the WEB_URL field, Taranis-AI autonomously determines the content to be collected. Even though it is mostly reliable, sometimes it is not perfect.

Advanced configuration

When content cannot be reliably collected using the Basic configuration, adding the attribute XPATH (See tutorial how to find it), can be useful. It is crucial to specify the XPath of the precise element containing the desired data.

RSS alternative for Mastodon feeds

The native Mastodon Collector is the preferred option because it supports authenticated timelines, account lookup, pagination, and collection cursors. The RSS Collector remains available as a simpler alternative for public hashtag or account feeds exposed by an instance.

To collect a Mastodon RSS feed, follow these steps:

  1. Finding the Mastodon RSS Feed URL:

    • Hashtag Feed: Add .rss to the hashtag URL. For example, to collect posts tagged with #cybersecurity: https://mastodon.social/tags/cybersecurity.rss
    • User Feed: Similarly, add .rss to the user’s profile URL. Example: https://mastodon.social/@username.rss

  2. Creating a New RSS Source with Required Parameters: When creating the new RSS source, configure it with the following parameters. Here’s an example of how to fill out the fields:

    • FEED_URL: Enter the RSS feed URL for the Mastodon hashtag or user (e.g., https://mastodon.social/tags/cybersecurity.rss).
    • CONTENT_LOCATION Set this to "summary" to specify the main content location within each RSS entry.
    • REFRESH_INTERVAL Set the refresh interval in crontab-like style, see Bots - refresh_interval.
    • DIGEST_SPLITTING is set to "false" since we’re not splitting entries into multiple items.

Configuration for Darkweb Feeds

Extend your compose.yml with a tor service, e.g.

tor:
  image: "docker.io/dperson/torproxy:latest"
  deploy:
    restart_policy:
      condition: always
  environment:
    # LOCATION: "AT"
  logging:
    driver: "json-file"
    options:
      max-size: "200k"
      max-file: "10"

Read details about the used docker image here

The important setting is “PROXY_SERVER” in the OSINT Source you want to crawl.

RT Collector

RT Collector enables Taranis AI to collect data from a user-defined Request Tracker instance.

RT Collector collects tickets, translates all ticket attachments into individual News Items. A ticket is represented via a Story. It also collects ticket Custom Fields and saves it as key-value pairs represented with Story attributes, visible whilst Story editing. On each collector execution an update to existing Stories occurs, so editing the Story values in Taranis AI is not recommended and handling it more like read-only items is better.

  • Required fields:

    • BASE_URL: Base URL of the RT instance (e.g. http://localhost).
    • RT_TOKEN: User token for the RT instance.

  • Optional fields:

    • SEARCH_QUERY: query to use for filtering tickets (e.g. owner=‘user1’). It is possible to check this manually by: http://<rt_address>/REST/2.0/tickets?query=owner=‘user1’
    • FIELDS_TO_INCLUDE: case-sensitive, comma-separated values; example: Email, IP; if not set, all ticket custom fields are ingested
    • ADDITIONAL_HEADERS
    • TLP_LEVEL
    • USER_AGENT
    • PROXY_SERVER
    • USE_GLOBAL_PROXY

MISP Collector

Until the definitions of our MISP Objects are not officially part of the MISP platform, feel free to import them manually (see MISP Objects). This allows to edit the information of News Items and Story data directly in the MISP instance without Taranis AI.

MISP Collector enables Taranis AI to collect MISP events.

  • Required fields:

    • URL: Base URL to the MISP instance (e.g. https://localhost)
    • API_KEY: API key to access the instance (see MISP Automation API)

  • Optional fields:

    • SSL_CHECK: if enabled, the SSL certificate will be validated
    • SHARING_GROUP_ID: set to the ID of a sharing group, if only one sharing group should be collected (see Create and manage Sharing Groups).
    • REQUEST_TIMEOUT
    • USER_AGENT
    • PROXY_SERVER
    • USE_GLOBAL_PROXY
    • ADDITIONAL_HEADERS
    • REFRESH_INTERVAL

How MISP Collector works

Essentially it works exactly like other collectors with one exception: conflicts. Given the nature of the collaborative environment of MISP events (they can be changed in the MISP platform by the owning organisation and secondary organisations can submit change requests using the MISP proposals). Due to that, there will likely occur conflicts when attempting to update existing Stories that were, in the meantime, internally modified.

Generally, conflicts occur the moment, a Story is modified internally, and has not been pushed to MISP immediately. Therefore, it is recommended to always try to keep Stories in sync with the MISP events. To update them in MISP with the Story (see Connectors).

Digest Splitting

Digest Splitting is a feature that allows the user to split all available URLs in the located element into individual News Items. The Digest Splitting Limit is the maximum number of URLs that will be split into individual News Items. If the limit is reached, the remaining URLs are dropped. The Digest Splitting Limit is set to 30 News Items by default but can be adjusted by the administrator. Useful in case of timeouts during collection of too many News Items.

Browser Mode

Collectors will fail if the web page content is only available with JavaScript. In that case it is possible to turn on the Browser Mode. All requests will have JavaScript enabled, therefore, it is slower and can use more resources.

1 - Mastodon Collector

Collect Mastodon hashtag, home, and account timelines through the Mastodon API.

The Mastodon Collector imports posts from a Mastodon instance through scheduled API polling. It supports hashtag timelines, the access-token owner’s home timeline, and public posts from a specific account.

Use the native collector when you need authenticated access, reliable pagination, or collection progress across runs. For simple public feeds, the RSS alternative may be sufficient.

Timeline modes

TimelineCollected postsTarget fieldAccess tokenRecommended scopes
hashtagPosts visible in the configured instance’s hashtag timelineHASHTAGOptional only when the instance permits anonymous hashtag accessread:statuses when a token is required
homeThe token owner’s home timelineNoneRequiredprofile and read:statuses
accountPosts from a public account known to the configured instanceACCOUNTRequiredread:accounts and read:statuses

Mastodon instances decide whether anonymous hashtag access is allowed. A hashtag can therefore work without a token on one instance and require authentication on another. The instance’s federation state, moderation policy, and knowledge of remote accounts also determine which posts are available; an instance is not a complete index of the Fediverse.

See Mastodon’s API documentation for the hashtag and home timeline endpoints, account lookup and account-status endpoints, and OAuth scopes.

Create an OSINT source

  1. Open Administration → OSINT Source.
  2. Select New OSINT Source.
  3. Enter a name and select Mastodon Collector.
  4. Configure the parameters described below.
  5. Save the source.
  6. Use Preview to verify the result without adding posts to Assess.
  7. Use Collect when the preview is correct, or enable the source for scheduled collection.

Parameters

ParameterRequiredDescription
INSTANCE_URLYesOrigin of the Mastodon instance, for example https://chaos.social. Do not include credentials, a path, query, or fragment.
TIMELINEYesOne of hashtag, home, or account.
HASHTAGHashtag modeHashtag with or without a leading #, for example curl or #curl. Letters, numbers, and underscores are accepted.
ACCOUNTAccount modeAccount handle without a URL, for example alice@example.social. A leading @ is optional.
ACCESS_TOKENHome and account modes; sometimes hashtag modeUser access token created on the same Mastodon instance as INSTANCE_URL.
USER_AGENTNoUser-Agent sent to Mastodon. The default is TaranisAI/1.0.
PROXY_SERVERNoPer-source HTTP(S) proxy URL.
USE_GLOBAL_PROXYNoUses the default collector proxy instead of PROXY_SERVER.
TLP_LEVELNoTLP level assigned to imported News Items.
REFRESH_INTERVALNoFive-field cron schedule. If empty, the global collector interval applies. See collector scheduling.

The global Collector Entry Limit in Administration → Settings limits the number of posts processed during each run.

Example: #curl on chaos.social

ParameterValue
INSTANCE_URLhttps://chaos.social
TIMELINEhashtag
HASHTAGcurl
ACCESS_TOKENA token created on chaos.social with read:statuses

If the token field is empty and the instance restricts anonymous hashtag access, collection fails with the actionable access-token-required message described under Troubleshooting.

Create a Mastodon access token

Mastodon’s web interface places access-token management under Development, not under the normal account or privacy settings. The token must be created on the instance configured in INSTANCE_URL; a token from another instance will not work.

  1. Sign in to the Mastodon instance used by the source.
  2. Open Preferences → Development. The direct URL is normally https://INSTANCE/settings/applications. For example, use chaos.social Development for https://chaos.social.
  3. Select New application.
  4. Enter an application name such as Taranis AI.
  5. Leave Application website empty unless your organization wants to identify the deployment.
  6. Leave the default redirect URI urn:ietf:wg:oauth:2.0:oob. Taranis does not use an interactive OAuth redirect.
  7. Select only the scopes needed for the configured timeline:
    • Hashtag: read:statuses.
    • Home: profile and read:statuses.
    • Account: read:accounts and read:statuses.
  8. Submit the application.
  9. Open the created application and copy Your access token. Do not copy the client secret instead.
  10. Paste the token into the Taranis source’s ACCESS_TOKEN field and save the source.

Some Mastodon versions preselect profile on the new-application form. Clear it for hashtag or account collection when it is not listed above. For one application used by all three modes, select profile, read:accounts, and read:statuses. Do not select write, follow, push, or any admin scope. On Mastodon versions older than 4.3, use read:accounts instead of profile for home-timeline collection.

Mastodon’s developer documentation describes user tokens, granular scopes, and the full OAuth authorization flow. The Development page is the shortest setup path for a Taranis administrator who controls the Mastodon account.

Token security

  • Treat the access token like a password. Do not paste it into tickets, chat, screenshots, shell history, or logs.
  • Prefer a dedicated organizational Mastodon account rather than a personal account for production collection.
  • Grant only the scopes listed for the selected timeline. The collector never needs write or administrator access.
  • Taranis masks the stored secret after saving. Revealing or replacing it is an audited administrator action.
  • If a token may have leaked, delete or revoke the Mastodon application, create a replacement, and update the Taranis source.

Collection behavior

The collector polls Mastodon’s REST API; it does not open a long-lived streaming connection and does not update Mastodon timeline markers.

On the first successful run, Taranis imports the newest posts up to the global Collector Entry Limit. Later runs request posts newer than the saved cursor. The cursor is stored with the latest collector task result and advances only after the collected News Items are published successfully. A failed Mastodon request or failed publication does not advance it.

Cursor state is intentionally best-effort. Deleting task history, retaining no task result for an inactive source, or restoring a database without recent task results can remove it. The next run then starts again from the newest posts. Core deduplication makes replay safe, but posts older than the entry limit may be skipped after such a reset.

Changing the instance, timeline mode, hashtag, token owner, or target account establishes a new timeline identity and starts a new cursor. Preview ignores the cursor and never changes collection progress.

Mapping and deduplication

  • Each Mastodon post becomes a News Item.
  • Boosts are stored as the original post and deduplicate through the original post URL.
  • Replies and boosts visible in the selected timeline are retained.
  • A content warning becomes the title when present; otherwise Taranis derives the title from the post content.
  • Media descriptions provide content when a post has no text.
  • The source account, author, publication time, language, post URL, and plain-text content are retained when supplied by Mastodon.

See Mastodon’s Status entity for the upstream response fields.

Troubleshooting

Access token required

Message: This Mastodon instance requires an access token to collect hashtags

The instance does not permit the combined hashtag feed to be read anonymously. Create a token with read:statuses, add it to the source, and preview again. This is expected on instances that expose local hashtag posts publicly but require authentication for remote posts.

Mastodon documents this behavior on the hashtag timeline endpoint.

Authentication failed or access was denied

Verify that:

  • The token was created on exactly the same instance as INSTANCE_URL.
  • The token was copied from Your access token, not Client secret.
  • The application has the scopes required for the selected mode.
  • The application or token has not been revoked.
  • The Mastodon account is still active and permitted to use the timeline.

Timeline or account not found

  • Enter a hashtag without a URL. A leading # is optional.
  • Enter an account as user@example.social, not as a profile URL.
  • Confirm that the account is visible to and known by the instance in INSTANCE_URL.
  • Open the hashtag or account from the instance’s own web interface to confirm that it exists there.

Rate limit exceeded

Increase REFRESH_INTERVAL or the global collector interval. Mastodon applies rate limits per account and IP address; see the Mastodon rate-limit documentation.

Instance unavailable

Check DNS, TLS certificates, firewall rules, the configured proxy, and whether the instance API is reachable from the collector worker. INSTANCE_URL must be the instance origin, not a profile, hashtag, or API URL.

No new statuses

This is a normal not-modified result. It means the timeline contains no posts newer than the saved cursor, or all returned posts already exist in Taranis.

Official Mastodon references