{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true, "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Delete this cell to re-enable tracebacks\n", "import sys\n", "ipython = get_ipython()\n", "\n", "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n", " exception_only=False, running_compiled_code=False):\n", " etype, value, tb = sys.exc_info()\n", " return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n", "\n", "ipython.showtraceback = hide_traceback" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "collapsed": true, "nbsphinx": "hidden" }, "outputs": [], "source": [ "# JSON output syntax highlighting\n", "from __future__ import print_function\n", "from pygments import highlight\n", "from pygments.lexers import JsonLexer\n", "from pygments.formatters import HtmlFormatter\n", "from IPython.display import HTML\n", "\n", "original_print = print\n", "\n", "def json_print(inpt):\n", " string = str(inpt)\n", " if string[0] == '{':\n", " formatter = HtmlFormatter()\n", " return HTML('{}'.format(\n", " formatter.get_style_defs('.highlight'),\n", " highlight(string, JsonLexer(), formatter)))\n", " else:\n", " original_print(inpt)\n", "\n", "print = json_print" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# without this configuration, only last print() call is outputted in cells\n", "from IPython.core.interactiveshell import InteractiveShell\n", "InteractiveShell.ast_node_interactivity = \"all\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# DataStore API\n", "\n", "CTI Python STIX2 features a new interface for pulling and pushing STIX2 content. The new interface consists of [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin), [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) and [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) constructs: a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) for pulling STIX2 content, a [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) for pushing STIX2 content, and a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) for both pulling and pushing.\n", "\n", "The DataStore, [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource), [DataSink](../api/stix2.datastore.rst#stix2.datastore.DataSink) (collectively referred to as the \"DataStore suite\") APIs are not referenced directly by a user but are used as base classes, which are then subclassed by real DataStore suites. CTI Python STIX2 provides the DataStore suites of [FileSystem](../api/datastore/stix2.datastore.filesystem.rst), [Memory](../api/datastore/stix2.datastore.memory.rst), and [TAXII](../api/datastore/stix2.datastore.taxii.rst). Users are also encouraged to subclass the base classes and create their own custom DataStore suites." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## CompositeDataSource\n", "\n", "[CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) is an available controller that can be used as a single interface to a set of defined [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). The purpose of this controller is allow for the grouping of [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) and making `get()`/`query()` calls to a set of DataSources in one API call. [CompositeDataSources](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) can be used to organize/group [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource), federate `get()`/`all_versions()`/`query()` calls, and reduce user code.\n", "\n", "[CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) is just a wrapper around a set of defined [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) (e.g. [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource)) that federates `get()`/`all_versions()`/`query()` calls individually to each of the attached [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) , collects the results from each [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) and returns them.\n", "\n", "Filters can be attached to [CompositeDataSources](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) just as they can be done to [DataStores](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) and [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). When `get()`/`all_versions()`/`query()` calls are made to the [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource), it will pass along any query filters from the call and any of its own filters to the attached [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource). In addition, those [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource) may have their own attached filters as well. The effect is that all the filters are eventually combined when the `get()`/`all_versions()`/`query()` call is actually executed within a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource). \n", "\n", "A [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) can also be attached to a [CompositeDataSource](../api/stix2.datastore.rst#stix2.datastore.CompositeDataSource) for multiple layers of grouped [DataSources](../api/stix2.datastore.rst#stix2.datastore.DataSource).\n", "\n", "\n", "### CompositeDataSource API\n", "\n", "#### CompositeDataSource Examples\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "intrusion-set",\n",
       "    "id": "intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a",\n",
       "    "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",\n",
       "    "created": "2017-05-31T21:31:53.197Z",\n",
       "    "modified": "2017-05-31T21:31:53.197Z",\n",
       "    "name": "DragonOK",\n",
       "    "description": "DragonOK is a threat group that has targeted Japanese organizations with phishing emails. Due to overlapping TTPs, including similar custom tools, DragonOK is thought to have a direct or indirect relationship with the threat group Moafee. [[Citation: Operation Quantum Entanglement]][[Citation: Symbiotic APT Groups]] It is known to use a variety of malware, including Sysget/HelloBridge, PlugX, PoisonIvy, FormerFirstRat, NFlog, and NewCT. [[Citation: New DragonOK]]",\n",
       "    "aliases": [\n",
       "        "DragonOK"\n",
       "    ],\n",
       "    "external_references": [\n",
       "        {\n",
       "            "source_name": "mitre-attack",\n",
       "            "url": "https://attack.mitre.org/wiki/Group/G0017",\n",
       "            "external_id": "G0017"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "Operation Quantum Entanglement",\n",
       "            "description": "Haq, T., Moran, N., Vashisht, S., Scott, M. (2014, September). OPERATION QUANTUM ENTANGLEMENT. Retrieved November 4, 2015.",\n",
       "            "url": "https://www.fireeye.com/content/dam/fireeye-www/global/en/current-threats/pdfs/wp-operation-quantum-entanglement.pdf"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "Symbiotic APT Groups",\n",
       "            "description": "Haq, T. (2014, October). An Insight into Symbiotic APT Groups. Retrieved November 4, 2015.",\n",
       "            "url": "https://dl.mandiant.com/EE/library/MIRcon2014/MIRcon%202014%20R&D%20Track%20Insight%20into%20Symbiotic%20APT.pdf"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "New DragonOK",\n",
       "            "description": "Miller-Osborn, J., Grunzweig, J.. (2015, April). Unit 42 Identifies New DragonOK Backdoor Malware Deployed Against Japanese Targets. Retrieved November 4, 2015.",\n",
       "            "url": "http://researchcenter.paloaltonetworks.com/2015/04/unit-42-identifies-new-dragonok-backdoor-malware-deployed-against-japanese-targets/"\n",
       "        }\n",
       "    ],\n",
       "    "object_marking_refs": [\n",
       "        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
{\n",
       "    "type": "indicator",\n",
       "    "id": "indicator--02b90f02-a96a-43ee-88f1-1e87297941f2",\n",
       "    "created": "2017-11-13T07:00:24.000Z",\n",
       "    "modified": "2017-11-13T07:00:24.000Z",\n",
       "    "name": "Ransomware IP Blocklist",\n",
       "    "description": "IP Blocklist address from abuse.ch",\n",
       "    "pattern": "[ ipv4-addr:value = '91.237.247.24' ]",\n",
       "    "valid_from": "2017-11-13T07:00:24Z",\n",
       "    "labels": [\n",
       "        "malicious-activity",\n",
       "        "Ransomware",\n",
       "        "Botnet",\n",
       "        "C&C"\n",
       "    ],\n",
       "    "external_references": [\n",
       "        {\n",
       "            "source_name": "abuse.ch",\n",
       "            "url": "https://ransomwaretracker.abuse.ch/blocklist/"\n",
       "        }\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from taxii2client import Collection\n", "from stix2 import CompositeDataSource, FileSystemSource, TAXIICollectionSource\n", "\n", "# create FileSystemStore\n", "fs = FileSystemSource(\"/home/michael/cti-python-stix2/stix2/test/stix2_data/\")\n", "\n", "# create TAXIICollectionSource\n", "colxn = Collection('https://test.freetaxii.com:8000/osint/collections/a9c22eaf-0f3e-482c-8bb4-45ae09e75d9b/')\n", "ts = TAXIICollectionSource(colxn)\n", "\n", "# add them both to the CompositeDataSource\n", "cs = CompositeDataSource()\n", "cs.add_data_sources([fs,ts])\n", "\n", "# get an object that is only in the filesystem\n", "intrusion_set = cs.get('intrusion-set--f3bdec95-3d62-42d9-a840-29630f6cdc1a')\n", "print(intrusion_set)\n", "\n", "# get an object that is only in the TAXII collection\n", "ind = cs.get('indicator--02b90f02-a96a-43ee-88f1-1e87297941f2')\n", "print(ind)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Filters\n", "\n", "The CTI Python STIX2 DataStore suites - [FileSystem](../api/datastore/stix2.datastore.filesystem.rst), [Memory](../api/datastore/stix2.datastore.memory.rst), and [TAXII](../api/datastore/stix2.datastore.taxii.rst) - all use the [Filters](../api/datastore/stix2.datastore.filters.rst) module to allow for the querying of STIX content. The basic functionality is that filters can be created and supplied everytime to calls to `query()`, and/or attached to a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) so that every future query placed to that [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) is evaluated against the attached filters, supplemented with any further filters supplied with the query call. Attached filters can also be removed from [DataStores](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin).\n", "\n", "Filters are very simple, as they consist of a field name, comparison operator and an object property value (i.e. value to compare to). All properties of STIX2 objects can be filtered on. In addition, TAXII2 Filtering parameters for fields can also be used in filters.\n", "\n", "TAXII2 filter fields:\n", "\n", "* added_after\n", "* match[id]\n", "* match[type]\n", "* match[version]\n", "\n", "Supported operators:\n", "\n", "* =\n", "* !=\n", "* in\n", "* >\n", "* < \n", "* ```>=```\n", "* <=\n", "\n", "Value types of the property values must be one of these (Python) types:\n", "\n", "* bool\n", "* dict\n", "* float\n", "* int\n", "* list\n", "* str\n", "* tuple\n", "\n", "### Filter Examples" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import sys\n", "from stix2 import Filter\n", "\n", "# create filter for STIX objects that have external references to MITRE ATT&CK framework\n", "f = Filter(\"external_references.source_name\", \"=\", \"mitre-attack\")\n", "\n", "# create filter for STIX objects that are not of SDO type Attack-Pattnern\n", "f1 = Filter(\"type\", \"!=\", \"attack-pattern\")\n", "\n", "# create filter for STIX objects that have the \"threat-report\" label\n", "f2 = Filter(\"labels\", \"in\", \"threat-report\")\n", "\n", "# create filter for STIX objects that have been modified past the timestamp\n", "f3 = Filter(\"modified\", \">=\", \"2017-01-28T21:33:10.772474Z\")\n", "\n", "# create filter for STIX objects that have been revoked\n", "f4 = Filter(\"revoked\", \"=\", True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For Filters to be applied to a query, they must be either supplied with the query call or attached to a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin), more specifically to a [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) whether that [DataSource](../api/stix2.datastore.rst#stix2.datastore.DataSource) is a part of a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) or stands by itself. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from stix2 import MemoryStore, FileSystemStore, FileSystemSource\n", "\n", "fs = FileSystemStore(\"/home/michael/Desktop/sample_stix2_data\")\n", "fs_source = FileSystemSource(\"/home/michael/Desktop/sample_stix2_data\")\n", "\n", "# attach filter to FileSystemStore\n", "fs.source.filters.add(f)\n", "\n", "# attach multiple filters to FileSystemStore\n", "fs.source.filters.update([f1,f2])\n", "\n", "# can also attach filters to a Source\n", "# attach multiple filters to FileSystemSource\n", "fs_source.filters.update([f3, f4])\n", "\n", "\n", "mem = MemoryStore()\n", "\n", "# As it is impractical to only use MemorySink or MemorySource,\n", "# attach a filter to a MemoryStore\n", "mem.source.filters.add(f)\n", "\n", "# attach multiple filters to a MemoryStore\n", "mem.source.filters.update([f1,f2])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## De-Referencing Relationships\n", "\n", "Given a STIX object, there are several ways to find other STIX objects related to it. To illustrate this, let's first create a [DataStore](../api/stix2.datastore.rst#stix2.datastore.DataStoreMixin) and add some objects and relationships." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "from stix2 import Campaign, Identity, Indicator, Malware, Relationship\n", "\n", "mem = MemoryStore()\n", "cam = Campaign(name='Charge', description='Attack!')\n", "idy = Identity(name='John Doe', identity_class=\"individual\")\n", "ind = Indicator(labels=['malicious-activity'], pattern=\"[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']\")\n", "mal = Malware(labels=['ransomware'], name=\"Cryptolocker\", created_by_ref=idy)\n", "rel1 = Relationship(ind, 'indicates', mal,)\n", "rel2 = Relationship(mal, 'targets', idy)\n", "rel3 = Relationship(cam, 'uses', mal)\n", "mem.add([cam, idy, ind, mal, rel1, rel2, rel3])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If a STIX object has a `created_by_ref` property, you can use the [creator_of()](../api/stix2.datastore.rst#stix2.datastore.DataSource.creator_of) method to retrieve the [Identity](../api/stix2.v20.sdo.rst#stix2.v20.sdo.Identity) object that created it." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "identity",\n",
       "    "id": "identity--be3baac0-9aba-48a8-81e4-4408b1c379a8",\n",
       "    "created": "2017-11-21T22:14:45.213Z",\n",
       "    "modified": "2017-11-21T22:14:45.213Z",\n",
       "    "name": "John Doe",\n",
       "    "identity_class": "individual"\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(mem.creator_of(mal))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use the [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) method to retrieve all the relationship objects that reference a STIX object." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rels = mem.relationships(mal)\n", "len(rels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can limit it to only specific relationship types:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Relationship(type='relationship', id='relationship--bd6fd399-c907-4feb-b1da-b90f15942f1d', created='2017-11-21T22:14:45.214Z', modified='2017-11-21T22:14:45.214Z', relationship_type=u'indicates', source_ref='indicator--5ee33ff0-c50d-456b-a8dd-b5d1b69a66e8', target_ref='malware--66c0bc78-4e27-4d80-a565-a07e6eb6fba4')]" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem.relationships(mal, relationship_type='indicates')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can limit it to only relationships where the given object is the source:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Relationship(type='relationship', id='relationship--7eb7f5cd-8bf2-4f7c-8756-84c0b5693b9a', created='2017-11-21T22:14:45.215Z', modified='2017-11-21T22:14:45.215Z', relationship_type=u'targets', source_ref='malware--66c0bc78-4e27-4d80-a565-a07e6eb6fba4', target_ref='identity--be3baac0-9aba-48a8-81e4-4408b1c379a8')]" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem.relationships(mal, source_only=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And you can limit it to only relationships where the given object is the target:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Relationship(type='relationship', id='relationship--bd6fd399-c907-4feb-b1da-b90f15942f1d', created='2017-11-21T22:14:45.214Z', modified='2017-11-21T22:14:45.214Z', relationship_type=u'indicates', source_ref='indicator--5ee33ff0-c50d-456b-a8dd-b5d1b69a66e8', target_ref='malware--66c0bc78-4e27-4d80-a565-a07e6eb6fba4'),\n", " Relationship(type='relationship', id='relationship--3c759d40-c92a-430e-aab6-77d5c5763302', created='2017-11-21T22:14:45.215Z', modified='2017-11-21T22:14:45.215Z', relationship_type=u'uses', source_ref='campaign--82ab7aa4-d13b-4e99-8a09-ebcba30668a7', target_ref='malware--66c0bc78-4e27-4d80-a565-a07e6eb6fba4')]" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem.relationships(mal, target_only=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, you can retrieve all STIX objects related to a given STIX object using [related_to()](../api/stix2.datastore.rst#stix2.datastore.DataSource.related_to). This calls [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) but then performs the extra step of getting the objects that these Relationships point to. [related_to()](../api/stix2.datastore.rst#stix2.datastore.DataSource.related_to) takes all the same arguments that [relationships()](../api/stix2.datastore.rst#stix2.datastore.DataSource.relationships) does." ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Campaign(type='campaign', id='campaign--82ab7aa4-d13b-4e99-8a09-ebcba30668a7', created='2017-11-21T22:14:45.213Z', modified='2017-11-21T22:14:45.213Z', name=u'Charge', description=u'Attack!')]" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mem.related_to(mal, target_only=True, relationship_type='uses')" ] } ], "metadata": { "kernelspec": { "display_name": "cti-python-stix2", "language": "python", "name": "cti-python-stix2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.12" } }, "nbformat": 4, "nbformat_minor": 2 }