Enterprise Open Source Intelligence Gathering – Part 2 Blogs, Message Boards and Metadata

This post is part two of my three part series on Enterprise Open Source Intelligence Gathering.  This information relates to the presentation that I am giving this week at the 7th Annual Ohio Information Security Summit.  For more background information, see part 1.  Part three will be about putting together a simple monitoring program/toolkit and creating a Internet postings (social media) policy for your company.

Part one of the series discussed ways to gather OSINT on social networks and some of the challenges this creates.  Besides gathering OSINT on social networks there are many more sources of information that company information may be posted on.  These include blogs, message boards and document repositories.  One of the byproducts of finding documents is metadata, which I will explain in more detail below.

OSINT and Blogs
Blogs can be searched via any traditional search engine, however, the challenge with blogs are not necessarily the posts themselves but the comments.  When it comes to blog posts the comments are usually where the action is, especially when it comes to your current and former employees (even customers) commenting on highly sensitive pubic relations issues that a company might be conducting damage control over.  The other point to make about commenting is that employees might be posting things that be violating one of your policies and cause brand reputation problems.  Examples of this are all the countless leaks of profits, downsizing, confidential information and more that the news media reports on.  Wouldn’t be great to be monitoring blogs and their comments to find these things out before they go viral?

Listed below are some of the blog and comment search sites that I recommend you add to your monitoring arsenal which I will talk about creating in part three:

Social Mention http://socialmention.com (has *great* comment search and RSS for monitoring)
Google Blog Search http://blogsearch.google.com (great for creating RSS feeds and very customizable)
Blogpulse http://www.blogpulse.com/ (has comment search)
Technorati http://technorati.com/
IceRocket http://www.icerocket.com/
BackType http://www.backtype.com/ (has comment search)
coComment http://www.cocomment.com/ (has comment search)

OSINT and Message Boards
Message boards have always been a great source of OSINT.  Message boards date back before blogs were popular and are still widely used today.  Because there are so many message boards out there that could contain good OSINT you really need to use message board search engines unless you know about specific message boards that you know your employees use (or could).  Good examples of these are job related message boards like vault.com or Yahoo/Google Finance discussion forums or groups centered around stock trading.

Here is my list of message board search engines and a few that might be more specific for a company:

Google Groups http://groups.google.com/ (always a good choice for creating RSS feeds and very customizable)
Yahoo! Groups http://groups.yahoo.com/
Big Boards http://www.big-boards.com/ (huge list!)
BoardReader http://boardreader.com/ (very good search and RSS feeds of results)
Board Tracker http://boardtracker.com/ (very good search and RSS feeds of results)

More specific:
Craigslist Forums http://www.craigslist.org/about/sites (RSS available)
Vault www.vault.com (job/employee discussions)
Google Finance http://www.google.com/finance (search for company stock symbol and check out the discussions)
XSSed http://www.xssed.com/ (XSS security vulnerabilities)
Full Disclosure Mailing List http://seclists.org/fulldisclosure/ (Security vulnerability disclosure)

Document Repositories
Something that I have seen more of recently are sites called document repositories.  These sites either aggregate documents found from various sources on the Internet or people can upload their own documents and presentations for public sharing purposes.  These sites are probably my favorite since you will find all sorts of interesting information!  Here is my list of favorites:

Docstoc http://www.docstoc.com/
*Really good document search engine.  I wish there was better RSS for it but they have an API in which Yahoo! Pipes could probably be used.

Scribd http://www.scribd.com/ (RSS feed of results)
SlideShare http://www.slideshare.net/ (RSS feed of results)
PDF Search Engine http://www.pdf-search-engine.com/
Toodoc http://www.toodoc.com/

Great! You found documents.  Now what?
Once you find interesting documents be sure to check out the document metadata.  What is metadata? Metadata is simply “data about data”.  Metadata in documents is traditionally used for indexing files as well as finding out information about the document creator and what software was used to create the document.  It goes without saying that document metadata is a treasure trove of information that could be used against your company.  For example, vulnerable versions of software that can be used for client side attacks, OS versions, path disclosure, user id’s and more can all be viewed through document metadata.

There are lots of good tools to pull out metadata from documents and pictures. With some of these tools it’s even possible to write a script to automatically strip metadata from documents and pictures (start with the script Larry Pesce wrote in his SANS paper below).  However, the best method for removing metadata in my opinion is to make sure it’s removed (or limited) in the first place!  If you are creating a new document make sure you are removing it or not allowing the application to save some of the more revealing things like user id’s and OS/version numbers.  If you want more detail on metadata and how to use some of the tools that are available check out the great paper over at the SANS InfoSec Reading Room titled “Document Metadata, the Silent Killer created by Larry Pesce.  Here is a short list of tools I use (or have used) to analyze metadata:

EXIFtool http://www.sno.phy.queensu.ca/~phil/exiftool/ (my personal favorite! The swiss army knife of metadata tools)
Metagoofil http://www.edge-security.com/metagoofil.php
Maltego (built-in metadata transform) http://www.paterva.com/web4/index.php/maltego (another favorite!)
Meta-Extractor http://meta-extractor.sourceforge.net/
FOCA http://www.informatica64.com/foca/

What’s the deal with brand reputation?
One last point I want to make is about brand reputation.  You may ask yourself, how does brand reputation relate to information security? Why should we care?  I have found it interesting that many of us in information security have been asked to do more research on brand reputation issues because no one else in the company had those types of skill sets to monitor information.  Brand reputation is vital to an organization, even more so in this economy.  Think of the CIA triad…Confidentiality, Integrity and Availability.  All three have aspects that reflect brand reputation.  All of us in information security need to be thinking of brand reputation in our daily job.

Next up in part three
In part three I will talk about setting up a simple monitoring program with the sites and tools I have mentioned thus far.  This will include how to start using Yahoo! Pipes to aggregate many of the feeds I talked about.  I will also conclude with information on how to create a Internet Postings Policy or now better known as a Social Media Policy for your company and why this is more important then ever.