{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Delete this cell to re-enable tracebacks\n", "import sys\n", "ipython = get_ipython()\n", "\n", "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n", " exception_only=False, running_compiled_code=False):\n", " etype, value, tb = sys.exc_info()\n", " return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n", "\n", "ipython.showtraceback = hide_traceback" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# JSON output syntax highlighting\n", "from __future__ import print_function\n", "from pygments import highlight\n", "from pygments.lexers import JsonLexer, TextLexer\n", "from pygments.formatters import HtmlFormatter\n", "from IPython.display import display, HTML\n", "from IPython.core.interactiveshell import InteractiveShell\n", "\n", "InteractiveShell.ast_node_interactivity = \"all\"\n", "\n", "def json_print(inpt):\n", " string = str(inpt)\n", " formatter = HtmlFormatter()\n", " if string[0] == '{':\n", " lexer = JsonLexer()\n", " else:\n", " lexer = TextLexer()\n", " return HTML('{}'.format(\n", " formatter.get_style_defs('.highlight'),\n", " highlight(string, lexer, formatter)))\n", "\n", "globals()['print'] = json_print" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## FileSystem \n", "\n", "The FileSystem suite contains [FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore), [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource) and [FileSystemSink](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSink). Under the hood, all FileSystem objects point to a file directory (on disk) that contains STIX 2 content. \n", "\n", "The directory and file structure of the intended STIX 2 content should be:\n", "\n", "```\n", "stix2_content/\n", " /STIX2 Domain Object type\n", " STIX2 Domain Object\n", " STIX2 Domain Object\n", " .\n", " .\n", " .\n", " /STIX2 Domain Object type\n", " STIX2 Domain Object\n", " STIX2 Domain Object\n", " .\n", " .\n", " .\n", " .\n", " .\n", " .\n", " /STIX2 Domain Object type\n", "```\n", "\n", "The master STIX 2 content directory contains subdirectories, each of which aligns to a STIX 2 domain object type (i.e. \"attack-pattern\", \"campaign\", \"malware\", etc.). Within each STIX 2 domain object subdirectory are JSON files that are STIX 2 domain objects of the specified type. The name of the json files correspond to the ID of the STIX 2 domain object found within that file. A real example of the FileSystem directory structure:\n", "\n", "```\n", "stix2_content/\n", " /attack-pattern\n", " attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6.json\n", " attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22.json\n", " attack-pattern--1b7ba276-eedc-4951-a762-0ceea2c030ec.json\n", " /campaign\n", " /course-of-action\n", " course-of-action--2a8de25c-f743-4348-b101-3ee33ab5871b.json\n", " course-of-action--2c3ce852-06a2-40ee-8fe6-086f6402a739.json\n", " /identity\n", " identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5.json\n", " /indicator\n", " /intrusion-set\n", " /malware\n", " malware--1d808f62-cf63-4063-9727-ff6132514c22.json\n", " malware--2eb9b131-d333-4a48-9eb4-d8dec46c19ee.json\n", " /observed-data\n", " /report\n", " /threat-actor\n", " /vulnerability\n", "```\n", "\n", "[FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore) is intended for use cases where STIX 2 content is retrieved and pushed to the same file directory. As [FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore) is just a wrapper around a paired [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource) and [FileSystemSink](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSink) that point the same file directory.\n", "\n", "For use cases where STIX 2 content will only be retrieved or pushed, then a [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource) and [FileSystemSink](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSink) can be used individually. They can also be used individually when STIX 2 content will be retrieved from one distinct file directory and pushed to another.\n", "\n", "### FileSystem API\n", "\n", "A note on [get()](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource.get), [all_versions()](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource.all_versions), and [query()](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource.query): The format of the STIX2 content targeted by the FileSystem suite is JSON files. When the [FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore) retrieves STIX 2 content (in JSON) from disk, it will attempt to parse the content into full-featured python-stix2 objects and returned as such. \n", "\n", "A note on [add()](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSink.add): When STIX content is added (pushed) to the file system, the STIX content can be supplied in the following forms: Python STIX objects, Python dictionaries (of valid STIX objects or Bundles), JSON-encoded strings (of valid STIX objects or Bundles), or a (Python) list of any of the previously listed types. Any of the previous STIX content forms will be converted to a STIX JSON object (in a STIX Bundle) and written to disk. \n", "\n", "### FileSystem Examples\n", "\n", "#### FileSystemStore\n", "\n", "Use the FileSystemStore when you want to both retrieve STIX content from the file system and push STIX content to it, too." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "malware",\n",
       "    "id": "malware--00c3bfcb-99bd-4767-8c03-b08f585f5c8a",\n",
       "    "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",\n",
       "    "created": "2017-05-31T21:33:19.746Z",\n",
       "    "modified": "2017-05-31T21:33:19.746Z",\n",
       "    "name": "PowerDuke",\n",
       "    "description": "PowerDuke is a backdoor that was used by APT29 in 2016. It has primarily been delivered through Microsoft Word or Excel attachments containing malicious macros.[[Citation: Volexity PowerDuke November 2016]]",\n",
       "    "labels": [\n",
       "        "malware"\n",
       "    ],\n",
       "    "external_references": [\n",
       "        {\n",
       "            "source_name": "mitre-attack",\n",
       "            "url": "https://attack.mitre.org/wiki/Software/S0139",\n",
       "            "external_id": "S0139"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "Volexity PowerDuke November 2016",\n",
       "            "description": "Adair, S.. (2016, November 9). PowerDuke: Widespread Post-Election Spear Phishing Campaigns Targeting Think Tanks and NGOs. Retrieved January 11, 2017.",\n",
       "            "url": "https://www.volexity.com/blog/2016/11/09/powerduke-post-election-spear-phishing-campaigns-targeting-think-tanks-and-ngos/"\n",
       "        }\n",
       "    ],\n",
       "    "object_marking_refs": [\n",
       "        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import FileSystemStore\n", "\n", "# create FileSystemStore\n", "fs = FileSystemStore(\"/tmp/stix2_store\")\n", "\n", "# retrieve STIX2 content from FileSystemStore\n", "ap = fs.get(\"attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6\")\n", "mal = fs.get(\"malware--00c3bfcb-99bd-4767-8c03-b08f585f5c8a\")\n", "\n", "# for visual purposes\n", "print(mal)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from stix2 import ThreatActor, Indicator\n", "\n", "# create new STIX threat-actor\n", "ta = ThreatActor(name=\"Adjective Bear\",\n", " labels=[\"nation-state\"],\n", " sophistication=\"innovator\",\n", " resource_level=\"government\",\n", " goals=[\n", " \"compromising media outlets\",\n", " \"water-hole attacks geared towards political, military targets\",\n", " \"intelligence collection\"\n", " ])\n", "\n", "# create new indicators\n", "ind = Indicator(description=\"Crusades C2 implant\",\n", " labels=[\"malicious-activity\"],\n", " pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "ind1 = Indicator(description=\"Crusades C2 implant 2\",\n", " labels=[\"malicious-activity\"],\n", " pattern=\"[file:hashes.'SHA-256' = '64c7e05e40a59511743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "# add STIX object (threat-actor) to FileSystemStore\n", "fs.add(ta)\n", "\n", "# can also add multiple STIX objects to FileSystemStore in one call\n", "fs.add([ind, ind1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### FileSystemSource\n", "\n", "Use the FileSystemSource when you only want to retrieve STIX content from the file system." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "attack-pattern",\n",
       "    "id": "attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6",\n",
       "    "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",\n",
       "    "created": "2017-05-31T21:30:54.176Z",\n",
       "    "modified": "2017-05-31T21:30:54.176Z",\n",
       "    "name": "Indicator Removal from Tools",\n",
       "    "description": "If a malicious...command-line parameters, Process monitoring",\n",
       "    "kill_chain_phases": [\n",
       "        {\n",
       "            "kill_chain_name": "mitre-attack",\n",
       "            "phase_name": "defense-evasion"\n",
       "        }\n",
       "    ],\n",
       "    "external_references": [\n",
       "        {\n",
       "            "source_name": "mitre-attack",\n",
       "            "url": "https://attack.mitre.org/wiki/Technique/T1066",\n",
       "            "external_id": "T1066"\n",
       "        }\n",
       "    ],\n",
       "    "object_marking_refs": [\n",
       "        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import FileSystemSource\n", "\n", "# create FileSystemSource\n", "fs_source = FileSystemSource(\"/tmp/stix2_source\")\n", "\n", "# retrieve STIX 2 objects\n", "ap = fs_source.get(\"attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6\")\n", "\n", "# for visual purposes\n", "print(ap)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
malware--96b08451-b27a-4ff6-893f-790e26393a8e\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--b42378e0-f147-496f-992a-26a49705395b\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--92ec0cbd-2c30-44a2-b270-73f4ec949841\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import Filter\n", "\n", "# create filter for type=malware\n", "query = [Filter(\"type\", \"=\", \"malware\")]\n", "\n", "# query on the filter\n", "mals = fs_source.query(query)\n", "\n", "for mal in mals:\n", " print(mal.id)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
malware--92ec0cbd-2c30-44a2-b270-73f4ec949841\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# add more filters to the query\n", "query.append(Filter(\"modified\", \">\" , \"2017-05-31T21:33:10.772474Z\"))\n", "\n", "mals = fs_source.query(query)\n", "\n", "# for visual purposes\n", "for mal in mals:\n", " print(mal.id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### FileSystemSink\n", "\n", "Use the FileSystemSink when you only want to push STIX content to the file system." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from stix2 import FileSystemSink, Campaign, Indicator\n", "\n", "# create FileSystemSink\n", "fs_sink = FileSystemSink(\"/tmp/stix2_sink\")\n", "\n", "# create STIX objects and add to sink\n", "camp = Campaign(name=\"The Crusades\",\n", " objective=\"Infiltrating Israeli, Iranian and Palestinian digital infrastructure and government systems.\",\n", " aliases=[\"Desert Moon\"])\n", "\n", "ind = Indicator(description=\"Crusades C2 implant\",\n", " labels=[\"malicious-activity\"],\n", " pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "ind1 = Indicator(description=\"Crusades C2 implant\",\n", " labels=[\"malicious-activity\"],\n", " pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "# add Campaign object to FileSystemSink\n", "fs_sink.add(camp)\n", "\n", "# can also add STIX objects to FileSystemSink in on call\n", "fs_sink.add([ind, ind1])" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }