{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Delete this cell to re-enable tracebacks\n", "import sys\n", "ipython = get_ipython()\n", "\n", "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n", " exception_only=False, running_compiled_code=False):\n", " etype, value, tb = sys.exc_info()\n", " value.__cause__ = None # suppress chained exceptions\n", " return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n", "\n", "ipython.showtraceback = hide_traceback" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# JSON output syntax highlighting\n", "from __future__ import print_function\n", "from pygments import highlight\n", "from pygments.lexers import JsonLexer, TextLexer\n", "from pygments.formatters import HtmlFormatter\n", "from IPython.display import display, HTML\n", "from IPython.core.interactiveshell import InteractiveShell\n", "\n", "InteractiveShell.ast_node_interactivity = \"all\"\n", "\n", "def json_print(inpt):\n", " string = str(inpt)\n", " formatter = HtmlFormatter()\n", " if string[0] == '{':\n", " lexer = JsonLexer()\n", " else:\n", " lexer = TextLexer()\n", " return HTML('{}'.format(\n", " formatter.get_style_defs('.highlight'),\n", " highlight(string, lexer, formatter)))\n", "\n", "globals()['print'] = json_print" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## FileSystem \n", "\n", "The FileSystem suite contains [FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore), [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource) and [FileSystemSink](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSink). Under the hood, all FileSystem objects point to a file directory (on disk) that contains STIX 2 content. \n", "\n", "The directory and file structure of the intended STIX 2 content should be:\n", "\n", "```\n", "stix2_content/\n", " STIX2 Domain Object type/\n", " STIX2 Domain Object ID/\n", " 'modified' timestamp.json\n", " 'modified' timestamp.json\n", " STIX2 Domain Object ID/\n", " 'modified' timestamp.json\n", " .\n", " .\n", " STIX2 Domain Object type/\n", " STIX2 Domain Object ID/\n", " 'modified' timestamp.json\n", " .\n", " .\n", " .\n", " .\n", " .\n", " .\n", " STIX2 Domain Object type/\n", "```\n", "\n", "The master STIX 2 content directory contains subdirectories, each of which aligns to a STIX 2 domain object type (i.e. \"attack-pattern\", \"campaign\", \"malware\", etc.). Within each STIX 2 domain object type's subdirectory are further subdirectories containing JSON files that are STIX 2 domain objects of the specified type; the name of each of these subdirectories is the ID of the associated STIX 2 domain object. Inside each of these subdirectories are JSON files, the names of which correspond to the ``modified`` timestamp of the STIX 2 domain object found within that file. A real example of the FileSystem directory structure:\n", "\n", "```\n", "stix2_content/\n", " /attack-pattern\n", " /attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6\n", " 20201211035036648071.json\n", " /attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22\n", " 20201210035036648071.json\n", " /attack-pattern--1b7ba276-eedc-4951-a762-0ceea2c030ec\n", " 20201111035036648071.json\n", " /campaign\n", " /course-of-action\n", " /course-of-action--2a8de25c-f743-4348-b101-3ee33ab5871b\n", " 20201011035036648071.json\n", " /course-of-action--2c3ce852-06a2-40ee-8fe6-086f6402a739\n", " 20201010035036648071.json\n", " /identity\n", " /identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5\n", " 20201215035036648071.json\n", " /indicator\n", " /intrusion-set\n", " /malware\n", " /malware--1d808f62-cf63-4063-9727-ff6132514c22\n", " 20201211045036648071.json\n", " /malware--2eb9b131-d333-4a48-9eb4-d8dec46c19ee\n", " 20201211035036648072.json\n", " /observed-data\n", " /report\n", " /threat-actor\n", " /vulnerability\n", "```\n", "\n", "[FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore) is intended for use cases where STIX 2 content is retrieved and pushed to the same file directory. As [FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore) is just a wrapper around a paired [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource) and [FileSystemSink](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSink) that point the same file directory.\n", "\n", "For use cases where STIX 2 content will only be retrieved or pushed, then a [FileSystemSource](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource) and [FileSystemSink](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSink) can be used individually. They can also be used individually when STIX 2 content will be retrieved from one distinct file directory and pushed to another.\n", "\n", "### FileSystem API\n", "\n", "A note on [get()](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource.get), [all_versions()](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource.all_versions), and [query()](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSource.query): The format of the STIX2 content targeted by the FileSystem suite is JSON files. When the [FileSystemStore](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemStore) retrieves STIX 2 content (in JSON) from disk, it will attempt to parse the content into full-featured python-stix2 objects and returned as such. \n", "\n", "A note on [add()](../api/datastore/stix2.datastore.filesystem.rst#stix2.datastore.filesystem.FileSystemSink.add): When STIX content is added (pushed) to the file system, the STIX content can be supplied in the following forms: Python STIX objects, Python dictionaries (of valid STIX objects or Bundles), JSON-encoded strings (of valid STIX objects or Bundles), or a (Python) list of any of the previously listed types. Any of the previous STIX content forms will be converted to a STIX JSON object (in a STIX Bundle) and written to disk. \n", "\n", "### FileSystem Examples\n", "\n", "#### FileSystemStore\n", "\n", "Use the FileSystemStore when you want to both retrieve STIX content from the file system and push STIX content to it, too." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "malware",\n",
       "    "spec_version": "2.1",\n",
       "    "id": "malware--92ec0cbd-2c30-44a2-b270-73f4ec949841",\n",
       "    "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",\n",
       "    "created": "2017-05-31T21:33:26.565Z",\n",
       "    "modified": "2017-05-31T21:33:26.565Z",\n",
       "    "name": "RTM",\n",
       "    "description": "RTM is custom malware written in Delphi. It is used by the group of the same name (RTM).[[Citation: ESET RTM Feb 2017]]",\n",
       "    "malware_types": [\n",
       "        "malware"\n",
       "    ],\n",
       "    "is_family": false,\n",
       "    "external_references": [\n",
       "        {\n",
       "            "source_name": "mitre-attack",\n",
       "            "url": "https://attack.mitre.org/wiki/Software/S0148",\n",
       "            "external_id": "S0148"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "ESET RTM Feb 2017",\n",
       "            "description": "Faou, M. and Boutin, J.. (2017, February). Read The Manual: A Guide to the RTM Banking Trojan. Retrieved March 9, 2017.",\n",
       "            "url": "https://www.welivesecurity.com/wp-content/uploads/2017/02/Read-The-Manual.pdf"\n",
       "        }\n",
       "    ],\n",
       "    "object_marking_refs": [\n",
       "        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import FileSystemStore\n", "\n", "# create FileSystemStore\n", "fs = FileSystemStore(\"/tmp/stix2_store\")\n", "\n", "# retrieve STIX2 content from FileSystemStore\n", "ap = fs.get(\"attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22\")\n", "mal = fs.get(\"malware--92ec0cbd-2c30-44a2-b270-73f4ec949841\")\n", "\n", "# for visual purposes\n", "print(mal.serialize(pretty=True))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from stix2 import ThreatActor, Indicator\n", "\n", "# create new STIX threat-actor\n", "ta = ThreatActor(name=\"Adjective Bear\",\n", " sophistication=\"innovator\",\n", " resource_level=\"government\",\n", " goals=[\n", " \"compromising media outlets\",\n", " \"water-hole attacks geared towards political, military targets\",\n", " \"intelligence collection\"\n", " ])\n", "\n", "# create new indicators\n", "ind = Indicator(description=\"Crusades C2 implant\",\n", " pattern_type=\"stix\",\n", " pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "ind1 = Indicator(description=\"Crusades C2 implant 2\",\n", " pattern_type=\"stix\",\n", " pattern=\"[file:hashes.'SHA-256' = '64c7e05e40a59511743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "# add STIX object (threat-actor) to FileSystemStore\n", "fs.add(ta)\n", "\n", "# can also add multiple STIX objects to FileSystemStore in one call\n", "fs.add([ind, ind1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### FileSystemSource\n", "\n", "Use the FileSystemSource when you only want to retrieve STIX content from the file system." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    "type": "attack-pattern",\n",
       "    "spec_version": "2.1",\n",
       "    "id": "attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22",\n",
       "    "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",\n",
       "    "created": "2017-05-31T21:30:19.735Z",\n",
       "    "modified": "2017-05-31T21:30:19.735Z",\n",
       "    "name": "Credential Dumping",\n",
       "    "description": "Credential dumping is the process of obtaining account login and password information from the operating system and software. Credentials can be used to perform Windows Credential Editor, Mimikatz, and gsecdump. These tools are in use by both professional security testers and adversaries.\\n\\nPlaintext passwords can be obtained using tools such as Mimikatz to extract passwords stored by the Local Security Authority (LSA). If smart cards are used to authenticate to a domain using a personal identification number (PIN), then that PIN is also cached as a result and may be dumped.Mimikatz access the LSA Subsystem Service (LSASS) process by opening the process, locating the LSA secrets key, and decrypting the sections in memory where credential details are stored. Credential dumpers may also use methods for reflective DLL Injection to reduce potential indicators of malicious activity.\\n\\nNTLM hash dumpers open the Security Accounts Manager (SAM) on the local file system (%SystemRoot%/system32/config/SAM) or create a dump of the Registry SAM key to access stored account password hashes. Some hash dumpers will open the local file system as a device and parse to the SAM table to avoid file access defenses. Others will make an in-memory copy of the SAM table before reading hashes. Detection of compromised Legitimate Credentials in-use by adversaries may help as well. \\n\\nOn Windows 8.1 and Windows Server 2012 R2, monitor Windows Logs for LSASS.exe creation to verify that LSASS started as a protected process.\\n\\nMonitor processes and command-line arguments for program execution that may be indicative of credential dumping. Remote access tools may contain built-in features or incorporate existing tools like Mimikatz. PowerShell scripts also exist that contain credential dumping functionality, such as PowerSploit's Invoke-Mimikatz module,[[Citation: Powersploit]] which may require additional logging features to be configured in the operating system to collect necessary information for analysis.\\n\\nPlatforms: Windows Server 2003, Windows Server 2008, Windows Server 2012, Windows XP, Windows 7, Windows 8, Windows Server 2003 R2, Windows Server 2008 R2, Windows Server 2012 R2, Windows Vista, Windows 8.1\\n\\nData Sources: API monitoring, Process command-line parameters, Process monitoring, PowerShell logs",\n",
       "    "kill_chain_phases": [\n",
       "        {\n",
       "            "kill_chain_name": "mitre-attack",\n",
       "            "phase_name": "credential-access"\n",
       "        }\n",
       "    ],\n",
       "    "external_references": [\n",
       "        {\n",
       "            "source_name": "mitre-attack",\n",
       "            "url": "https://attack.mitre.org/wiki/Technique/T1003",\n",
       "            "external_id": "T1003"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "Github Mimikatz Module sekurlsa",\n",
       "            "description": "Delpy, B. (2014, September 14). Mimikatz module ~ sekurlsa. Retrieved January 10, 2016.",\n",
       "            "url": "https://github.com/gentilkiwi/mimikatz/wiki/module-~-sekurlsa"\n",
       "        },\n",
       "        {\n",
       "            "source_name": "Powersploit",\n",
       "            "description": "PowerSploit. (n.d.).  Retrieved December 4, 2014.",\n",
       "            "url": "https://github.com/mattifestation/PowerSploit"\n",
       "        }\n",
       "    ],\n",
       "    "object_marking_refs": [\n",
       "        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import FileSystemSource\n", "\n", "# create FileSystemSource\n", "fs_source = FileSystemSource(\"/tmp/stix2_source\")\n", "\n", "# retrieve STIX 2 objects\n", "ap = fs_source.get(\"attack-pattern--0a3ead4e-6d47-4ccb-854c-a6a4f9d96b22\")\n", "\n", "# for visual purposes\n", "print(ap)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
malware--92ec0cbd-2c30-44a2-b270-73f4ec949841\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--b42378e0-f147-496f-992a-26a49705395b\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--96b08451-b27a-4ff6-893f-790e26393a8e\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stix2 import Filter\n", "\n", "# create filter for type=malware\n", "query = [Filter(\"type\", \"=\", \"malware\")]\n", "\n", "# query on the filter\n", "mals = fs_source.query(query)\n", "\n", "for mal in mals:\n", " print(mal.id)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
malware--92ec0cbd-2c30-44a2-b270-73f4ec949841\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/html": [ "
malware--6b616fc1-1505-48e3-8b2c-0d19337bff38\n",
       "
\n" ], "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# add more filters to the query\n", "query.append(Filter(\"modified\", \">\" , \"2017-05-31T21:33:10.772474Z\"))\n", "\n", "mals = fs_source.query(query)\n", "\n", "# for visual purposes\n", "for mal in mals:\n", " print(mal.id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### FileSystemSink\n", "\n", "Use the FileSystemSink when you only want to push STIX content to the file system." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "from stix2 import FileSystemSink, Campaign, Indicator\n", "\n", "# create FileSystemSink\n", "fs_sink = FileSystemSink(\"/tmp/stix2_sink\")\n", "\n", "# create STIX objects and add to sink\n", "camp = Campaign(name=\"The Crusades\",\n", " objective=\"Infiltrating Israeli, Iranian and Palestinian digital infrastructure and government systems.\",\n", " aliases=[\"Desert Moon\"])\n", "\n", "ind = Indicator(description=\"Crusades C2 implant\",\n", " pattern_type=\"stix\",\n", " pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "ind1 = Indicator(description=\"Crusades C2 implant\",\n", " pattern_type=\"stix\",\n", " pattern=\"[file:hashes.'SHA-256' = '54b7e05e39a59428743635242e4a867c932140a999f52a1e54fa7ee6a440c73b']\")\n", "\n", "# add Campaign object to FileSystemSink\n", "fs_sink.add(camp)\n", "\n", "# can also add STIX objects to FileSystemSink in one call\n", "fs_sink.add([ind, ind1])" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" } }, "nbformat": 4, "nbformat_minor": 2 }