Administration

• 1: User Management
• 2: OSINT workflow
• 3: OSINTSources
• 4: Report Types
• 5: Bots
• 6: Collectors
• 7: Publishers
• 8: Product Types
• 9: OpenAPI

1 - User Management

Organizations

Organizations can be added with following parameters defining them: name, description, address.

Roles

Define roles with certain permissions.

Users

Manage users and assign them a role, permissions and an organization.

Screenshots

Add new Organization

Edit basic user role

Add new User

2 - OSINT workflow

Functionalities

1. Create OSINT Sources
2. Add word lists
3. Adapt word list’s functionality
4. Enable include/exclude list filtering
5. Bot selection
6. Collect Sources

1. Create OSINT Sources

• Import/Export: Sources can be imported and exported as JSON
• CRUD: Sources can be created, updated and deleted. For each source a collector, feed URL and the content location can be defined among other things.

2. Add word lists

• Import/Export: Word lists can be imported and exported as json
• CRUD: Word lists can be created, updated and deleted.

3. Adapt word list’s functionality

Word lists can have the following functionalities (displayed under “usage”):

• Collector Includelist: Collected news items using words of this word list will be accepted
• Collector Excludelist: Collected news items using words of this word list will not be accepted
• Tagging Bot: Collected news items will be tagged with words from these word lists
• Collector Includelist & Tagging Bot: A word list can be used for tagging and include listing

4. Enable include/exclude list filtering

To activate include or exclude lists, they need to be added to the default source group.

It has to be mentioned, that this include/exclude filtering happens during the news item collection. Therefore, only filtered news items will be stored in the database and displayed in “Assess”.

5. Bot selection

After the collection, it is possible to adapt news items.

Therefore, following bots are currently available:

• Wordlist bot: Tags news items by wordlist
• IOC bot: Finds indicators of compromise in news items
• NLP tagging bot: Tags news items via NLP
• Story bot: Applies story clustering to news items
• Summary bot: Summarizes stories

CRUD: Bots can be created, updated and deleted.

Index: Decides the order of bots

RUN_AFTER_COLLECTOR: Indicates if bot is active after collection

6. Collect Sources

After all settings are made, sources can be collected. Either collect all sources by clicking on the “collect sources” button, or collect single sources.

3 - OSINTSources

Functionalities

Sources for gathering data are set in the OSINTSources. It is possible to:

• Create a new source
• Import sources
• Export sources
• Collect from sources

Create a new source

1. Select New Item to open editor at the bottom.
2. Enter a Name and, optionally, a Description.
3. Select desired collector. (See Collectors)

Import sources

Select Import and choose desired JSON file for import. (See Initial Setup)

Export sources

Select Export to download a JSON file containing your established collectors.

Collect from sources

Select Collect Sources to aggregate information from all established OSINT sources.

4 - Report Types

Attributes

Desired attributes need to be created first. Then they can be managed by the admin user. Besides name, description and default value also type, validator and validator parameter can be set.

Report Types - CRUD

• Create: Report types can be created (“New item”). After adding a new attribute group, different attributes can be added to this group.
• Read & Update: Report types (including their attribute groups) can be updated by clicking on them in the list.
• Delete: One or multiple reports can be deleted at once.

Screenshots

Add new Attribute

Report Types - Create new Report Type

Report Types - Add new Attribute Group

Report Types - Select new Attribute from list

5 - Bots

List of Bots

1. IOC BOT - for tagging news items
2. NLP Tagging BOT - for tagging news items via NLP
3. Story BOT - for story clustering
4. Summary BOT - for summarizing stories
5. Wordlist BOT tagging news items by wordlist

Bot’s settings

• Name
• Descrition
• Type: Select an option based on the desired functionalities.
• Index: Specifies the execution order of bots when RUN_AFTER_COLLECTOR is enabled.
• RUN_AFTER_COLLECTOR: Executes the bot after any collector.
• REFRESH_INTERVAL: Specifies the execution interval of the bot (default is 10 minutes).
  • Accepted values:
    • hourly
    • daily
    • weekly
    • number representing minutes (20 (run every 20 minutes))
    • a certain time (15:44) - run every day at set time (24h time format)

6 - Collectors

Supported options:

1. RSS Collector
2. Simple Web Collector
3. RT Collector

The administration view now allows users to use the Preview feature to see the result of the configuration without the items being processed further for the Assess view. This feature is available for RSS, Simple Web and RT collector.

RSS Collector

RSS Collector enables Taranis AI to collect data from a user-defined RSS feed (See RSS feeds details).

• Required fields:
  • FEED_URL
• Optional fields:
  • USER_AGENT
  • PROXY_SERVER
  • ADDITIONAL_HEADERS [accepts a valid json] (can be used to add additional headers, not all headers work as expected)
  • CONTENT_LOCATION
  • XPATH
  • TLP_LEVEL
  • REFRESH_INTERVAL (see Bots - refresh_interval)
  • DIGEST_SPLITTING On/Off (creates News Items out of URLs present in the Summary field of RSS feed)
  • DIGEST_SPLITTING_LIMIT (default: 30)
  • BROWSER_MODE On/Off (see Browser Mode)

Basic configuration

• Example:
  • FEED_URL: https://www.bsi.bund.de/SiteGlobals/Functions/RSSFeed/RSSNewsfeed/RSSNewsfeed.xml

Advanced configuration

The RSS Collector supports the use of XPath for locating elements. (See Simple Web Collector Advanced configuration)

• Example:
  • FEED_URL: https://www.bsi.bund.de/SiteGlobals/Functions/RSSFeed/RSSNewsfeed/RSSNewsfeed.xml
  • XPATH: //*[@id=“getting-started-web-console-creating-new-project_openshift-web-console”]
  • ADDITIONAL_HEADERS: { "AUTHORIZATION": "Bearer Token1234", "X-API-KEY": "12345", "Cookie": "firstcookie=1234; second-cookie=4321", }

Simple Web Collector

Simple Web Collector enables Taranis AI to collect data using web URLs and XPaths.

• Required field:
  • WEB_URL
• Optional fields:
  • USER_AGENT
  • PROXY_SERVER
  • ADDITIONAL_HEADERS
  • XPATH
  • TLP_LEVEL
  • DIGEST_SPLITTING On/Off
  • DIGEST_SPLITTING_LIMIT (default: 30)
  • BROWSER_MODE On/Off (see Browser Mode)

Basic configuration

The simplest way to use this collector is to use the WEB_URL field only. By using only the WEB_URL field, Taranis-AI autonomously determines the content to be collected. Even though it is mostly reliable, sometimes it is not perfect.

Advanced configuration

When content cannot be reliably collected using the Basic configuration, adding the attribute XPATH (See tutorial how to find it), can be useful. It is crucial to specify the XPath of the precise element containing the desired data.

RT Collector

RT Collector enables Taranis AI to collect data from a user-defined Request Tracker instance.

• Required fields:

  • BASE_URL: Base URL of the RT instance (e.g. localhost).
  • RT_TOKEN: User token for the RT instance.

• Optional fields:

  • ADDITIONAL_HEADERS
  • TLP_LEVEL

Digest Splitting

Digest Splitting is a feature that allows the user to split all available URLs in the located element into individual News Items. The Digest Splitting Limit is the maximum number of URLs that will be split into individual News Items. If the limit is reached, the remaining URLs are dropped. The Digest Splitting Limit is set to 30 News Items by default but can be adjusted by the administrator. Useful in case of timeouts during collection of too many News Items.

Browser Mode

Collectors will fail if the web page content is only available with JavaScript. In that case it is possible to turn on the Browser Mode. All requests will have JavaScript enabled, therefore, it is slower and can use more resources.

7 - Publishers

Supported options:

1. Email Publisher
2. FTP Publisher

Email Publisher

The Email Publisher allows sending out Products.

• Fields:
  • SMTP_SERVER_ADDRESS*: Address of the SMTP server.
  • SMTP_SERVER_PORT*: Port of the SMTP server.
  • SERVER_TLS: Enable/Disable TLS.
  • EMAIL_USERNAME: Login username for the SMTP server.
  • EMAIL_PASSWORD: Login password for the SMTP server.
  • EMAIL_SENDER*: Sender of the email for message envelope.
  • EMAIL_RECIPIENT*: Email address of the recipient for message envelope. It is possible to use only one email recipient.
  • EMAIL_SUBJECT: Subject of the email.

Note: The EMAIL_SENDER and EMAIL_RECIPIENT parameters are used to construct the message envelope used by the transport agents. Message headers are not modified by these parameters in any way.

Required fields are marked with a *.

General usage

Once the publisher is created, it becomes available in the “Publish” section of each product. To send out a product via email, the product must be “Rendered” first. To render a product, use the option available in the product’s view.

8 - Product Types

Prebuilt Product Types

• CERT Daily Report
• Default HTML Presenter
• Default MISP Presenter
• Default PDF Presenter
• Default TEXT Presenter

Editing Product Types

All crucial fields are editable, with the most important being Type, Template, and Report Types.

• Type: It’s the responsibility of the administrator to ensure the selected type is compatible with the subsequently provided template.
• Template: Users can select from prebuilt templates or add new ones.
• Report Types: This field determines which types of reports can be added to the products.

Create new Product Types

While there are several prebuilt product types available, users also have the option to create their own product types using custom templates.

It can be beneficial to create custom Product Types to meet desired results with the publishers.

Example of creating a simple new template using Jinja2

This is an example to render arbitrary values and loop over attributes.

1. Create a new file with a unique name in src/core/core/static/presenter_templates,
2. Write a custom template:

   TITLE: {{ data.report_items[0].get('title') | default('No title provided', true) }}<br>
   DATE CREATED: {{ data.report_items[0].get('created') | default('Not available', true) }}<br>
   LAST UPDATED: {{ data.report_items[0].get('last_updated') | default('Not available', true) }}<br>
   {% for name, attribute in data.report_items[0].get('attributes').items() %}
   {{ name }}: {{ attribute }}<br>
   {% endfor %}
   

4. Restart the Taranis AI instance.

Advanced behaviour

If needed, templates can be utilized for more complex renderings by leveraging custom attributes.

Currently, this functionality is demonstrated in the text_template.txt file, where the attribute omission of type “Omit Keys” allows for the exclusion of unnecessary attributes from publication. To employ this feature, the administrator simply needs to add this attribute to the relevant report type. Then, within a specific report (Analyze View), they can specify the attributes to omit by listing them as comma-separated strings.

It is essential to ensure that the “Name” used for the report type attribute matches exactly with the key used in the template.

9 - OpenAPI

Description

The admin user can access the Taranis API through Swagger UI. Swagger UI displays OpenAPI specifications as an interactive API documentation.

Functionalities 👤

see: Swagger UI

Screenshots

Taranis instance is alive