For the last two months, our engineering team has been hard at work rebuilding the architecture behind our lead discovery and engagement system.
With our new architecture, built on ElasticSearch, the gold standard search and analysis technology that powers Facebook, Salesforce, Netflix, and many others, Socedo will be able to find better leads for our customers faster than ever before.
What’s Changed with Our Lead Discovery System
Since our creation, Socedo has been powered by a homegrown infrastructure that took advantage of a concept called a “full text index.” Essentially, this “index” – just like the “index” you find at the back of a book – allows text data to be stored in a way that makes searching extremely fast. In our system, the leads we find on Twitter are like the pages of our book, and the tweets, following relationships, and enhanced bio and location information are the words.
Our old architecture utilized separate indexes for each of our customers’ campaigns, storing all the information about the leads and engagements in each campaign in a separate data store.
This allowed our homegrown infrastructure to scale as we’ve added more customers, but prevented any campaigns from leveraging the data discovered or learned by any other campaign.
ElasticSearch changes that by providing us a best-in-class platform we know can scale no matter what we throw at it. The critical difference is now all of the data we pull from Twitter is stored in a single giant “global index” so everyone can utilize public Twitter data (i.e. tweets, follower relationships) from all past searches in their own search to instantly get lead results.
In addition, if you have multiple campaigns, your own campaigns will be able to talk to each other so that you can customize how your campaigns engage with prospects based on the way other campaigns have already engaged with prospects.
Why ElasticSearch is Important
With this new “global index”, our system is able to re-use the data we’ve already pulled from Twitter to provide lead results more quickly. We’re always adding more and more data to our index and every customer can take advantage of this immediately.
Now you can completely swap your criteria to a brand new set of search terms, and because you’re piggy-backing on the data already in the index, you can get thousands of leads instantly.
It also makes criteria revision more powerful. In the new system, as soon as you update a keyword, you’ll be able to see the impacts of the update on the quality and quantity of your overall prospect pool immediately.
With ElasticSearch in place, you can also now change the way your campaigns engage with prospects based on the way other campaigns have already done so. For example, if you have multiple campaigns set up, you can ensure that those campaigns do not ever target the same leads or do not do so within a certain period of time.
In the future, we will be able to support even more sophisticated workflows where the engagements from a campaign can be customized based on previous interactions with the prospect.
In addition to the immediate changes to lead volume, search speed, and multiple campaign engagements, this new architecture is setting us up for exciting product innovations in the future. Here are some of the areas of innovation we’re focused on with our new global index:
Keyword Metrics & Suggestions
Using the shared data within ElasticSearch, we can start to provide a predictive understanding of how well keywords will perform for a campaign before they are actually used to search for leads.
We’re also working on algorithms to suggest additional keywords and keyword modifications to enable you to discover new leads based on the leads you’ve already converted and the performance of your existing keywords.
Combining Social with Your Existing Lead Data
ElasticSearch provides us the platform to ingest and correlate massive amounts of data, and it will enable a future where we can increase our intelligence based on existing data from your marketing automation platform and CRM system.
For example, we can inject information about your leads into our system to provide new insights on how to best engage your leads on social. For example, see what social topics are the most highly correlated with conversion or revenue. We can also take information about leads you’ve already called or met in person, and make sure your social outreach campaigns are more intelligent based on these offline interactions.
Adding more data sources to provide a richer picture of your prospects
Having a unified data storage system will enable us to continue to add on additional data sources and types of data in the near future. With this system, we could now add Facebook, Instagram, or other social network data alongside data from Twitter to provide a complete picture of a lead’s online presence. In this combined index our customers could query for leads that are talking about something on Twitter and fans of something else on Facebook.
We’ll also continue to leverage data from multiple sources to clean our lead records and provide the most up-to-date and complete picture of your prospects.
As the industry standard for search and analysis, ElasticSearch plugs into the most mature and well respected big data and machine learning platforms in the industry. With our new architecture, we’ll be able to take advantage of these platforms and the best-in-class algorithms to make our product even more intelligent at discovering and engaging with your prospects on social.
If you’re already using Socedo, now is a great time to re-visit your criteria keywords, log into your account to make a change to your criteria set to find better leads.
If you’re not using Socedo yet, start a trial today to find your target prospects on Twitter.