[UPDATE: January 2024]
Unfortunately, in 2023, the people at start.me changed how devices access the JSON files that drive their sites making our method of retrieving data ineffective.
Since the data in SMART was no longer accurate, I decided to shut the project down for now.
BLUF
I made a tool called SMART (Start Me Aggregated Resource Tool) at https://smart.myosint.training. It grabs data from all the OSINT-focused start.me sites and puts them in one place.
But why, Micah? Why?
I love that there are so many OSINT resources out there for us to use. From the osintframework.com site to all the start.me sites that have popped up in the past years, the amount of OSINT resources being shared is STAGGERING!
The problem? You have to visit every start.me to access their data. You cannot go to one site and just type in your search terms and see resources across ALL OSINT Start.me sites…..until SMART.
So what is SMART?
SMART (https://smart.myosint.training) is the web interface to a HUGE amount of data retrieved from over 300 OSINT-focused start.me pages. Behind the scenes, the data retrieval program crawls each public OSINT start.me, retrieving its resources and tagging them. Then, if the program finds a link to another start.me, it goes to that site and grabs that data…and so on.
This program takes minutes to run and outputs a file that has over 84,000 OSINT resources for you. You visit the SMART page and type in what you are looking for in the search field and you instantly have access to the data from all those start.me sites. Saves you time!
Are there caveats?
Of course there are:
- BE CAREFUL when visiting any of these links and resources. SMART is aggregating the resources and it is up to you to vet and validate the resources before using them. This is why the SMART tool does not have hyperlinks. This way you cannot accidentally click on a malicious link.
- SMART only grabs data from start.me pages. If your favorite OSINT person has a blog or other web site, their resources won’t be in here unless someone put those resources into a start.me.
- Are there old and bad links in there? Absolutely! Some start.me pages haven’t updated their resources in years (using SMART I found a start.me I put up in 2018 that I had forgotten about!).
- If an OSINT start.me page is not linked to other start.me pages, SMART will not find them. My suggestion is to get your site on https://start.me/p/Pwy0X4/osint-inception and then my tool will easily find it.
- Does this mean all those start.me pages are less useful? Oh no….quite the opposite. Without motivated OSINT enthusiasts making their start.me pages, SMART could not exist.
Anything interesting in the data?
Absolutely. I’ve found some great nuggets already. I’ll leave it up to you to find more.
Couldn’t someone use this for promotion?
As a recovering cyber security offensive security person, I always try to figure out how something could be weaponized or misused by people. For example, someone could create an OSINT start.me page that is linked to by others and is just stocked with URLs making advertisements and such. If I or people in the community point these out to me ([email protected]), I will block the site from being included in SMART.
SMART
The SMART tool has several user interface options you should know about:
- A disclaimer reminding you that there may be malicious data in it.
- A drop down list to choose how many results show on a page.
- The search field (with example “webbreacher” text).
- Icons you can click on to sort the columns.
Enjoy and please let me know if you find this useful or have suggestions to make it better.