Administration

1 - User Management

Organizations

Organizations can be added with following parameters defining them: name, description, address.

Roles

Define roles with permissions. Roles are then assigned to users to control access to administration, Assess, Analyze, Publish, connector, and worker features.

Users

Manage users and assign them roles and an organization.

The user table supports JSON import and export:

• Export downloads users_export.json. If users are selected, only those users are exported; otherwise all users are exported.
• The export format is version: 1 with a data list containing user names and usernames.
• Passwords are never exported.
• Import accepts the exported JSON format, then assigns the selected organization and selected roles to every imported user.
• Existing usernames are skipped and reported in a warning instead of failing the whole import.
• Imported users without a password stay passwordless. Use this for externally authenticated users, or set a local password afterwards.

Operational CLI

Operators can repair existing database-auth users from inside the core container with taranis-cli.

Reset an existing user’s password:

docker exec -it core taranis-cli set-password admin
kubectl exec -it deploy/core -- taranis-cli set-password admin

For non-interactive automation, pass the password through standard input:

printf '%s\n' 'new-password' | docker exec -i core taranis-cli set-password admin --password-stdin

Replace an existing user’s complete role list:

docker exec -it core taranis-cli set-roles user Admin
kubectl exec -it deploy/core -- taranis-cli set-roles user Admin User

set-roles accepts exact role names or role IDs. It replaces the full role list; include every role the user should keep. Prefer the password prompt or --password-stdin so passwords do not land in shell history.

Screenshots

Add new Organization

Edit basic user role

Add new User

2 - OSINT workflow

Functionalities

1. Create OSINT Sources
2. Add word lists
3. Adapt word list’s functionality
4. Enable include/exclude list filtering
5. Bot selection
6. Collect Sources

1. Create OSINT Sources

• Import/Export: Sources can be imported and exported as JSON
• CRUD: Sources can be created, updated and deleted. For each source a collector, feed URL and the content location can be defined among other things.

2. Add word lists

• Import/Export: Word lists can be imported and exported as json
• CRUD: Word lists can be created, updated and deleted.

3. Adapt word list’s functionality

Word lists can have the following functionalities (displayed under “usage”):

• Collector Includelist: Collected news items using words of this word list will be accepted
• Collector Excludelist: Collected news items using words of this word list will not be accepted
• Tagging Bot: Collected news items will be tagged with words from these word lists
• Collector Includelist & Tagging Bot: A word list can be used for tagging and include listing

4. Enable include/exclude list filtering

To activate include or exclude lists, they need to be added to the default source group.

It has to be mentioned, that this include/exclude filtering happens during the news item collection. Therefore, only filtered news items will be stored in the database and displayed in “Assess”.

5. Bot selection

After the collection, it is possible to adapt news items.

Common bots include:

• Wordlist Bot: Tags News Items by wordlist.
• IOC Bot: Finds indicators of compromise in News Items.
• NLP Bot: Tags News Items via named entity recognition.
• Story Clustering Bot: Groups related Stories.
• Summary Bot: Summarizes Stories and can generate Story titles.
• Sentiment Analysis Bot: Adds sentiment attributes to News Items.
• Cybersecurity Classifier Bot: Classifies whether content is cybersecurity related.

CRUD: Bots can be created, updated and deleted.

Index: Decides the order of bots

RUN_AFTER_COLLECTOR: Indicates if bot is active after collection

LLM-backed bots can use the optional LLM Bot Service. Full bot configuration is documented in Bots.

6. Collect Sources

After all settings are made, sources can be collected. Either collect all sources by clicking on the “collect sources” button, or collect single sources.

3 - OSINT Sources

Functionalities

OSINT Sources define where collectors gather data from. Administrators can:

• Create and edit sources.
• Enable or disable scheduled collection.
• Assign a source rank and icon.
• Preview collector output before ingesting it into Assess.
• Import and export source configuration as JSON.
• Collect one source or all sources on demand.

Create a new source

1. Select New Item to open editor at the bottom.
2. Enter a Name and, optionally, a Description.
3. Select a collector. See Collectors.
4. Configure collector parameters.
5. Save the source.

Use Source Rank to prioritize or classify sources for analyst workflows. Upload an icon when source identity should be visible on Story and News Item cards.

Preview a source

Use Preview from the source editor or table actions to run the collector without storing the result in Assess. Preview results are shown as Story cards. If a preview fails, use Retrigger preview after adjusting the source configuration.

Preview is especially useful for RSS, Simple Web, and RT sources where XPath, headers, proxies, or content extraction settings may need tuning.

Enable and disable sources

The source state button controls whether the source is active for scheduled collection. Disabling a source unschedules its Redis/RQ cron job; enabling it schedules the source again when it has a refresh interval.

Import sources

Select Import and choose the source JSON file. Imports support current exports and legacy source JSON formats.

Export sources

Select Export to download a JSON file. If sources are selected, only those sources are exported; otherwise all visible sources are exported.

Collect from sources

Use Collect to collect one source, or Collect All to collect all enabled sources. Collection is worker-backed; if the job is queued but does not run, verify worker health in Background Jobs.

4 - Report Types

Attributes

Desired attributes need to be created first. Then they can be managed by the admin user. Besides name, description and default value also type, validator and validator parameter can be set.

Report Types - CRUD

• Create: Report types can be created (“New item”). After adding a new attribute group, different attributes can be added to this group.
• Read & Update: Report types (including their attribute groups) can be updated by clicking on them in the list. Their update will not affect Reports, which already use the Report Type
• Delete: One or multiple reports can be deleted at once.

Screenshots

Add new Attribute

Report Types - Create new Report Type

Report Types - Add new Attribute Group

Report Types - Select new Attribute from list

5 - Bots

Bots run as worker jobs. They can be triggered manually, scheduled with REFRESH_INTERVAL, or run after collection when RUN_AFTER_COLLECTOR is enabled.

Available bots

Implementation details are available in the worker bot source.

Common settings

Scheduled bots are handled by Redis/RQ. See Background Jobs for worker and scheduler health checks.

LLM-backed bots

The optional LLM Bot Service can serve several bot endpoints.

Per-bot endpoint fields take precedence over worker environment defaults. After upgrading from older standalone bot services, update LLM-backed bot parameters that still point to summary_bot, nlp_bot, story_bot, or sentiment_analysis_bot. If you keep the Cybersecurity Classifier Bot, make sure its configured classifier endpoint is still deployed.

The Summary Bot only updates Story titles when TITLE_ENDPOINT is configured and the Story contains more than one News Item.

Sentiment output

The Sentiment Analysis Bot writes these News Item attributes:

• sentiment_score
• sentiment_category

The Story Edit advanced view displays the sentiment status when these attributes are present.

6 - Collectors

Supported options

1. RSS Collector
2. Simple Web Collector
3. RT Collector
4. MISP Collector

The administration view now allows users to use the Preview feature to see the result of the configuration without the items being processed further for the Assess view. This feature is available for RSS, Simple Web and RT collector.

RSS Collector

RSS Collector enables Taranis AI to collect data from a user-defined RSS feed (See RSS feeds details).

• Required fields:
  • FEED_URL
• Optional fields:
  • USER_AGENT
  • PROXY_SERVER
  • USE_GLOBAL_PROXY (ignore anything set in the PROXY_SERVER field, use what is currently set as a default in Settings; learn more in section Settings)
  • ADDITIONAL_HEADERS [accepts a valid json] (can be used to add additional headers, not all headers work as expected)
  • CONTENT_LOCATION (selects which field from the RSS entry should be used as the item content, for example description or content:encoded; when the collector uses content from the RSS entry, it first checks the field named here; if that field is empty or missing, it falls back through content, content:encoded, summary, and description)
  • USE_FEED_CONTENT (controls whether the collector uses content from the RSS entry itself or fetches the linked article page; if this is enabled, the collector uses RSS entry content; if it is disabled, the collector fetches the linked article page; if it is not set, a non-empty CONTENT_LOCATION effectively enables using RSS entry content)
  • XPATH (optional XPath expression used to limit the extracted content to a specific element; if USE_FEED_CONTENT is enabled, XPath is applied to the HTML content taken from the RSS entry; if the XPath does not match, the collector keeps the original RSS entry content; if USE_FEED_CONTENT is disabled, XPath is applied to the fetched article page; if the XPath does not match there, the collector does not fall back to another extraction method and the content may be empty)
  • TLP_LEVEL
  • REFRESH_INTERVAL (see Bots - refresh_interval)
  • DIGEST_SPLITTING On/Off (creates News Items out of URLs present in the Summary field of RSS feed)
  • DIGEST_SPLITTING_LIMIT (default: 30)
  • BROWSER_MODE On/Off (see Browser Mode)

Basic configuration

• Example:
  • FEED_URL: https://www.bmi.gv.at/rss/bmi_presse.xml

Advanced configuration

The RSS Collector supports the use of XPath for locating elements. (See Simple Web Collector Advanced configuration)

• Example:
  • FEED_URL: https://www.bmi.gv.at/rss/bmi_presse.xml
  • XPATH: //*[@id=“getting-started-web-console-creating-new-project_openshift-web-console”]
  • ADDITIONAL_HEADERS: { "AUTHORIZATION": "Bearer Token1234", "X-API-KEY": "12345", "Cookie": "firstcookie=1234; second-cookie=4321", }

Simple Web Collector

Simple Web Collector enables Taranis AI to collect data using web URLs and XPaths.

• Required field:
  • WEB_URL
• Optional fields:
  • USER_AGENT
  • PROXY_SERVER
  • USE_GLOBAL_PROXY
  • ADDITIONAL_HEADERS
  • XPATH (set to specify the location of the scraped element on the website)
  • TLP_LEVEL
  • DIGEST_SPLITTING On/Off
  • DIGEST_SPLITTING_LIMIT (default: 30)
  • BROWSER_MODE On/Off (see Browser Mode)

Basic configuration

The simplest way to use this collector is to use the WEB_URL field only. By using only the WEB_URL field, Taranis-AI autonomously determines the content to be collected. Even though it is mostly reliable, sometimes it is not perfect.

Advanced configuration

When content cannot be reliably collected using the Basic configuration, adding the attribute XPATH (See tutorial how to find it), can be useful. It is crucial to specify the XPath of the precise element containing the desired data.

Configuration for Mastodon Feeds

To set up an RSS Collector for collecting posts from a Mastodon hashtag or user, follow these steps:

1. Finding the Mastodon RSS Feed URL:

   • Hashtag Feed: Add .rss to the hashtag URL. For example, to collect posts tagged with #cybersecurity: https://mastodon.social/tags/cybersecurity.rss
   • User Feed: Similarly, add .rss to the user’s profile URL. Example: https://mastodon.social/@username.rss

2. Creating a New RSS Source with Required Parameters: When creating the new RSS source, configure it with the following parameters. Here’s an example of how to fill out the fields:

   • FEED_URL: Enter the RSS feed URL for the Mastodon hashtag or user (e.g., https://mastodon.social/tags/cybersecurity.rss).
   • CONTENT_LOCATION Set this to "summary" to specify the main content location within each RSS entry.
   • REFRESH_INTERVAL Set the refresh interval in crontab-like style, see Bots - refresh_interval.
   • DIGEST_SPLITTING is set to "false" since we’re not splitting entries into multiple items.

Configuration for Darkweb Feeds

Extend your compose.yml with a tor service, e.g.

tor:
  image: "docker.io/dperson/torproxy:latest"
  deploy:
    restart_policy:
      condition: always
  environment:
    # LOCATION: "AT"
  logging:
    driver: "json-file"
    options:
      max-size: "200k"
      max-file: "10"

Read details about the used docker image here

The important setting is “PROXY_SERVER” in the OSINT Source you want to crawl.

RT Collector

RT Collector enables Taranis AI to collect data from a user-defined Request Tracker instance.

RT Collector collects tickets, translates all ticket attachments into individual News Items. A ticket is represented via a Story. It also collects ticket Custom Fields and saves it as key-value pairs represented with Story attributes, visible whilst Story editing. On each collector execution an update to existing Stories occurs, so editing the Story values in Taranis AI is not recommended and handling it more like read-only items is better.

• Required fields:

  • BASE_URL: Base URL of the RT instance (e.g. http://localhost).
  • RT_TOKEN: User token for the RT instance.

• Optional fields:

  • SEARCH_QUERY: query to use for filtering tickets (e.g. owner=‘user1’). It is possible to check this manually by: http://<rt_address>/REST/2.0/tickets?query=owner=‘user1’
  • FIELDS_TO_INCLUDE: case-sensitive, comma-separated values; example: Email, IP; if not set, all ticket custom fields are ingested
  • ADDITIONAL_HEADERS
  • TLP_LEVEL
  • USER_AGENT
  • PROXY_SERVER
  • USE_GLOBAL_PROXY

MISP Collector

Until the definitions of our MISP Objects are not officially part of the MISP platform, feel free to import them manually (see MISP Objects). This allows to edit the information of News Items and Story data directly in the MISP instance without Taranis AI.

MISP Collector enables Taranis AI to collect MISP events.

• Required fields:

  • URL: Base URL to the MISP instance (e.g. https://localhost)
  • API_KEY: API key to access the instance (see MISP Automation API)

• Optional fields:

  • SSL_CHECK: if enabled, the SSL certificate will be validated
  • SHARING_GROUP_ID: set to the ID of a sharing group, if only one sharing group should be collected (see Create and manage Sharing Groups).
  • REQUEST_TIMEOUT
  • USER_AGENT
  • PROXY_SERVER
  • USE_GLOBAL_PROXY
  • ADDITIONAL_HEADERS
  • REFRESH_INTERVAL

How MISP Collector works

Essentially it works exactly like other collectors with one exception: conflicts. Given the nature of the collaborative environment of MISP events (they can be changed in the MISP platform by the owning organisation and secondary organisations can submit change requests using the MISP proposals). Due to that, there will likely occur conflicts when attempting to update existing Stories that were, in the meantime, internally modified.

Generally, conflicts occur the moment, a Story is modified internally, and has not been pushed to MISP immediately. Therefore, it is recommended to always try to keep Stories in sync with the MISP events. To update them in MISP with the Story (see Connectors).

Digest Splitting

Digest Splitting is a feature that allows the user to split all available URLs in the located element into individual News Items. The Digest Splitting Limit is the maximum number of URLs that will be split into individual News Items. If the limit is reached, the remaining URLs are dropped. The Digest Splitting Limit is set to 30 News Items by default but can be adjusted by the administrator. Useful in case of timeouts during collection of too many News Items.

Browser Mode

Collectors will fail if the web page content is only available with JavaScript. In that case it is possible to turn on the Browser Mode. All requests will have JavaScript enabled, therefore, it is slower and can use more resources.

7 - Connectors

Supported options:

• MISP Connector
  • Usage of the MISP Connector

MISP Connector

The MISP connector is currently in an experimental stage. Please submit issues when you discover any problems or need support with its usage.

Until the definitions of our MISP Objects are not officially part of the MISP platform, feel free to import them manually (see MISP Objects). This enables you to edit the objects of News Items and Story data directly in you MISP instances without Taranis AI.

MISP Connector enables Taranis AI to push Stories to MISP in the representation of MISP Events.

The created events contain automatically an Event Report based on the Story content (currently it encompasses the Story description and Story summary, if you feel the Event Report would benefit extending with more information, please open a feature request for it here).

First the user with administrative privileges needs to make sure, the permissions connected to Connectors usage are assigned appropriately, by default, they are not assigned to any user, not even the admins themselves. Then the Connector needs to be setup in the Admin section under the Connectors tab.

Explanation of individual permissions:

• Connector user access - ability to share a story to a Connector
• Config connector access - ability to access admin Connector configuration
• Config connector create - ability to create a Connector
• Config connector delete - ability to delete a Connector
• Config connector update - ability to update a Connector configuration

• Required parameters:

  • URL: Base URL to the MISP instance (e.g. https://localhost)
  • API_KEY: API key to access the instance (see MISP Automation API)
  • ORGANISATION_ID: ID of your organisation, usually starts with ID=1 (It is very important that this value is correct, otherwise the tool will not work properly!)

• Optional parameters:

  • SSL_CHECK: if enabled, the SSL certificate will be validated
  • SHARING_GROUP_ID: set to the ID of a sharing group, if you want to collect only one sharing group (see Create and manage Sharing Groups)
  • DISTRIBUTION: Integer value of the distribution of the created events (see Setting distribution and Creating an event)
  • REQUEST_TIMEOUT
  • USER_AGENT
  • PROXY_SERVER
  • ADDITIONAL_HEADERS
  • REFRESH_INTERVAL

  Usage of the MISP Connector

  To send a story to MISP, use the button “Share to Connector” in Story options in Assess. When an update to the Story is later on made, it is recommended to share the change immediately, you need to do this manually like the first time.

  Before a Story is pushed repeatedly to update the MISP event, it is important to check if any proposals are pending in the MISP event, as it would override the original content of it and would make it difficult to resolve the proposals correctly.

  

  When a Story is collected, that is not owned by your configured organisation, it is possible to update it in Taranis AI and by using the same “Share to Connector” button in the Story options a proposal to the event is created. Then the owning organisation should review this proposal, and approve or reject the changes.

8 - Publishers

Supported options

Publishers send rendered Products to external systems.

Keep API keys, passwords, app secrets, and private keys in protected configuration. Do not reuse personal credentials for shared publishers.

Email Publisher

The Email Publisher allows sending out Products.

• Fields:
  • SMTP_SERVER_ADDRESS*: Address of the SMTP server.
  • SMTP_SERVER_PORT*: Port of the SMTP server.
  • SERVER_TLS: Enable/Disable TLS.
  • EMAIL_USERNAME: Login username for the SMTP server.
  • EMAIL_PASSWORD: Login password for the SMTP server.
  • EMAIL_SENDER*: Sender of the email for message envelope.
  • EMAIL_RECIPIENT*: Email address of the recipient for message envelope. It is possible to use only one email recipient.
  • EMAIL_SUBJECT: Subject of the email.

Note: The EMAIL_SENDER and EMAIL_RECIPIENT parameters are used to construct the message envelope used by the transport agents. Message headers are not modified by these parameters in any way.

Required fields are marked with a *.

General usage

Once the publisher is created, it becomes available in the “Publish” section of each product. To send out a product via email, the product must be “Rendered” first. To render a product, use the option available in the product’s view.

Kafka Publisher

The Kafka Publisher sends each rendered Product as a JSON message to the configured topic.

• Message key: the generated product file name.
• Message body: object_name, mime_type, and data.
• KAFKA_SECURITY_PROTOCOL defaults to PLAINTEXT.
• Supported security protocols are PLAINTEXT and SASL_PLAINTEXT.
• When KAFKA_SECURITY_PROTOCOL is SASL_PLAINTEXT, set KAFKA_SASL_MECHANISM, KAFKA_SASL_USERNAME, and KAFKA_SASL_PASSWORD.
• KAFKA_ACKS defaults to all.
• KAFKA_RETRIES defaults to 3.
• KAFKA_SEND_TIMEOUT defaults to 30 seconds.

TAXII Publisher

The TAXII Publisher pushes STIX bundle objects to a TAXII 2.1 collection. The rendered Product must be valid JSON with type: bundle and a non-empty objects list.

Set AUTH_TYPE to basic with USERNAME and PASSWORD, or to bearer with API_TOKEN. Configure either TAXII_API_ROOT_URL directly or TAXII_DISCOVERY_URL so Taranis AI can discover the API root.

S3 Publisher

The S3 Publisher uploads the rendered Product to S3-compatible object storage. If the configured bucket does not exist, Taranis AI creates it before uploading.

9 - Product Types

Prebuilt Product Types

• CERT Daily Report
• Default HTML Presenter
• Default MISP Presenter
• Default PDF Presenter
• Default TEXT Presenter

Editing Product Types

All crucial fields are editable, with the most important being Type, Template, and Report Types.

• Type: It’s the responsibility of the administrator to ensure the selected type is compatible with the subsequently provided template.
• Template: Users can select from prebuilt templates or add new ones.
• Report Types: This field determines which types of reports can be added to the products.

Template paths are validated before use. A Product Type cannot render a Report Type unless that Report Type is associated with the Product Type.

Create new Product Types

While there are several prebuilt product types available, users also have the option to create their own product types using custom templates.

It can be beneficial to create custom Product Types to meet desired results with the publishers.

Example of creating a simple new template using Jinja2

This is an example to render arbitrary values and loop over attributes.

1. Create a new file with a unique name in src/core/core/static/presenter_templates.
2. Write a custom template:

   TITLE: {{ data.report_items[0].get('title') | default('No title provided', true) }}<br>
   DATE CREATED: {{ data.report_items[0].get('created') | default('Not available', true) }}<br>
   LAST UPDATED: {{ data.report_items[0].get('last_updated') | default('Not available', true) }}<br>
   {% for name, attribute in data.report_items[0].get('attributes').items() %}
   {{ name }}: {{ attribute }}<br>
   {% endfor %}
   

3. Select the new template in the Product Type.
4. Render a test Product before using the Product Type in production.

Advanced behaviour

If needed, templates can be utilized for more complex renderings by leveraging custom attributes.

Currently, this functionality is demonstrated in the text_template.txt file, where the attribute omission of type “Omit Keys” allows for the exclusion of unnecessary attributes from publication. To employ this feature, the administrator simply needs to add this attribute to the relevant report type. Then, within a specific report (Analyze View), they can specify the attributes to omit by listing them as comma-separated strings.

It is essential to ensure that the “Name” used for the report type attribute matches exactly with the key used in the template.

10 - Settings

Options

1. Default TLP Level
2. Default Collector Proxy
3. Default Collector Interval

Default TLP Level

Learn more about what TLP version 2.0 is under first.org/tlp/.

Default value: Clear

Options: Green, Amber, Amber+Strict, Red

All newly collected items by Collectors use this default TLP value. When this setting is changed, it does not affect the items that are already present in Assess section.

This setting can be overridden for each collector with the TLP_LEVEL field.

Default Collector Proxy

When a default collector proxy is set, it can then be used in the collectors by enabling the switch USE_GLOBAL_PROXY.

The following matrix of the states applies

• 🟢 Per-source proxy set: A proxy is set or not for a collector.
• 🔵 Default proxy set: A default proxy is set or not in global settings.
• USE_GLOBAL_PROXY: In the collector (OSINT Source) settings, the switch USE_GLOBAL_PROXY is enabled.
• Result: The resulting state.

Default Collector Interval

This value is used for REFRESH_INTERVAL for all collectors (OSINT Sources) when this field is left unset individually for all collectors. The syntax is the same as REFRESH_INTERVAL in bot settings.

Export options

• Export all Stories: Export all stories without any metadata (tags, etc.).
• Export all Stories with metadata: Export all stories with all the metadata (tags, attributes, likes, dislikes, relevance, etc.). It is not safe to use these data back for importing (some tags and attributes are used for internal purposes and could create unexpected behaviour). Drop metadata before importing, if doing so.

Danger Zone

• Clear all Worker Queues: Delete all messages from all worker queues. This action cannot be undone.
• Delete all Tags: Delete all tags from all Stories in the system. This action cannot be undone.

See Background Jobs before clearing worker queues.

11 - OpenAPI

Description

The admin user can access the Taranis API through Swagger UI. Swagger UI displays OpenAPI specifications as an interactive API documentation. Current OpenAPI specifications are available at https://github.com/taranis-ai/taranis-ai/blob/master/src/core/core/static/openapi3_1.yaml

Functionalities 👤

see: Swagger UI

Screenshots

Taranis instance is alive