Twitter Ends its Partnership with DataSift – Firehose Access Expires on August 13, 2015

Nick Halstead
11th April 2015 10 Comments


Our first and foremost concern upon receiving the news from Twitter about suspending our months-long negotiations and no further interest in being involved in PYLON, our new privacy-first technology, was to inform our customers and explain the situation. However, based on your feedback and what we’ve been reading in the press, I believe that more explanation is needed.

Thank you for all the words of support, but in case it hasn’t come across clearly enough – we’re fine. Our business model has never relied on access to Twitter data. We’ve built a robust big data processing platform, which is data source-agnostic, capable of dealing with billions of interactions a day from 20 other social and news networks we’re working with. And if you’ve heard any rumors about us extending our reach into business-owned data, well, they’re true.

This is also the very reason, we are now working with Facebook, the undisputed largest source of social data on earth, on providing the ecosystem with Facebook topic data. You will hear about topic data more in coming months. This data has never been available before and use cases for its application are practically limitless. It’s also a stepping stone for us in providing unprecedented access to social data, only possible now, because it’s delivered in a privacy-first model with aggregated and anonymized results.

Our goal has always been to provide a one-stop shop for our customers to access all the types of data from a variety of networks and be able to consume it in the most efficient way. Less noise, more actionable results. This is what truly matters to companies that deal with social data.

The bottom line: Twitter has seriously damaged the ecosystem this week. 80% of our customers use technology that can’t be replaced by Twitter. At the end of the day, Twitter is providing data licensing, not processing data to enable analysis.

Twitter also demonstrated that it doesn’t understand the basic rules of this market: social networks make money from engagement and advertising. Revenue from data should be a secondary concern to distribution and it should occur only in a privacy-safe way. Better understanding of their audiences means more engagement and more ad spend from brands. More noise = less ad spend.

I’ve been mentioning our ‘privacy-first’ approach throughout. This is the way forward for the new age of social data market as the universe of new networks, most of which deal solely with non-public data, grows massively. Social networks will come and go. As for DataSift, we will still be here.


For several months now, we’ve been working hard to renew our contract with Twitter. The negotiations were promising. At several points, we felt a deal was imminent. However, we no longer believe that a renewal of our partnership is likely.

DataSift’s customers will be able to access Twitter’s firehose of data as normal until August 13th, 2015. After that date all the customers will need to transition to other providers to receive Twitter data. This is an extremely disappointing result to us and the ecosystem of companies we have helped to build solutions around Twitter data.

In this blog, we want to provide some brief context to our partnership with Twitter, the ecosystem we’ve built together, and where we see the future for social data and DataSift.

How our partnership with Twitter started
By building a Big Data platform and APIs (instead of an end application) we have allowed our customers to use our platform as the foundation for their social analytics and intelligence solutions. Companies get the benefits of access to the full firehose of data from multiple social networks, with sophisticated data mining capabilities to separate signal from the noise.

89% percent of our customers use DataSift to access data from multiple social networks and we’ve expanded our data partnership to over 20 social networks and news providers. Most recently we announced a partnership with Facebook.

This core capability to provide a single, unified platform to access multiple networks is a major reason why customers love DataSift.

Today we power an ecosystem of 1,000 tech companies, enterprises and marketing agencies, many of whom embed DataSift into commercial applications that they sell to brands. The “customers of our customers” span into tens of thousands. This does not only benefit those brands, but also the social networks themselves.

An open ecosystem that reports independently of the network gives brands and advertisers more confidence in the data they are working with and allows the networks to concentrate on what we all love them for – providing us with the best, and most relevant, content. This is the foundation that Datasift is built on, it’s the approach that every social network has taken up to now and is the basis of our recently announced partnership with Facebook.

How DataSift’s Twitter license benefits our joint customers
With the end of our partnership with Twitter the disruption is not only measured by the impact on our 1,000 direct customers, but on the tens of thousands of companies that use applications that are “DataSift-powered”. Many of these companies create insights that drive direct advertising revenue back into Twitter. A direct switch to Twitter/GNIP will not mitigate that disruption. Today, 80% of our customers use our advanced processing capabilities that are not available from Twitter/GNIP. For example:

  • DataSift’s delivery and integration options help enterprises quickly and easily integrate Twitter data into their business intelligence platforms.
  • VEDO allows DataSift customers to automatically categorize social data based on its meaning, helping them to transform it into useful data.
  • DataSift not only unravels short links in Tweets, but also fetches the title and metadata of the link target.
  • With DataSift’s sentiment and demographics augmentations, brands can filter and categorize content based on favorability, age, gender, location and more.
  • DataSift does not impose any restrictions on the complexity of analysis that our clients can perform.
  • DataSift allows filtering on all data fields, meaning that clients can drill down to exactly the data they need.
  • The wildcard searching that DataSift offers makes building effective queries even simpler.
  • VEDO FOCUS is a powerful tool that allows our customers to capture Tweets related to specific industry sectors with one simple query.
  • DataSift automatically injects whitespace into Chinese and Japanese text to divide it into “tokens” before filtering.

Without these advanced capabilities, companies would be left with simple keyword searches which return large volumes of noisy and irrelevant data. This represents a step backwards for them, a step backwards for the industry and a step backwards for Twitter.

Smoothing the transition with a GNIP Connector
DataSift customers who wish to retain their Twitter feed will need to license their information from Twitter/GNIP directly. As such we feel it is important to make that transition as seamless as possible. Since the DataSift platform already has the technology to ingest data owned by the customer – we will now be opening up a GNIP connector so that as you make the transition, you can still bring Twitter data into the DataSift Platform. This will not fix many of the failings of the GNIP processing + filtering but it will still allow you to continue to use our API’s to receive the data.

The future of the social data market and DataSift
Our mission to enable companies to mine real-time data from a broad spectrum of Human Data (social, blog and news sources) remains unchanged. We continue to work with many social networks and to champion a privacy-first approach to the analysis of their data.

Our newly launched PYLON(TM) for Facebook Topic Data gives brands aggregated and anonymized insight into the opinions of the more than 1.39 billion people on Facebook. For the first time, aggregate analysis is enabled across the entirety of Facebook users – not just the public profiles. Access to topic data opens up an entire world of previously unavailable data that includes Non-Public Posts such as status updates, Page Posts, plus Comments, Likes and Shares on posts with demographics. We are now providing our clients with the most powerful tool in the world for understanding consumer opinion.

We will continue to sign up more networks, and provide better insights while protecting consumer information and trust. We look forward to continuing this journey with you.

If you have any questions regarding this, please get in touch with your sales representative or if you’re from the media, please email [email protected]

  • GabrielleNYC

    Gosh – Nick, that’s a blow.

  • Daniel Doherty

    Sorry to hear this Nick, but seems to be the way Twitter has been moving for a long time. I was hoping dataSift would continue being the exception due to the length and depth of relationship you had with Twitter. Great news that you will continue adding the data from other social networks. Best of luck for the future.

  • Ivan Mojsilovic

    I’m quite sure DataSift have and will have much better analytics tool than Twitter/Gnip. This move may lower the margins on DataSift’s revenue, but if Twitter thinks they will easily steal DataSift’s clients, they’re wrong. Good luck guys!

  • Everydaypanos

    Facebook is now the Best on Earth? #lol (Love how secrecy in the Business Model turns to bitch in Marketing)

  • gregorylent

    how you know twitter doesn’t know its own product.. the ceo doesn’t use it

  • PeterJ42

    Sorry to hear that Nick. If there’s anything we can do at DataScience Oxford to get the word out to our 500 DataScientists about the other platforms, the company data initiative etc., please let me know.

  • Watty Helms

    Twitter is revealing itself to be the money-grubbing, control freak that we always thought it would become post-IPO. But if you were to think that Facebook is any different you would be deluding yourself….

    • Nicholas Maietta

      I stay away from Facebook like the plague but have been a Twitter user for some time now. However, with the things that social networks have been doing with it’s user data, i may be leave social networks for good.

  • JR Lewis

    I am sorry for your loss.

  • Mike Rouse

    We use you for accessing Disqus data, so it just means I need to go to Twitter/GNIP directly to augment their data feeds into my app. However, I’d much prefer coming to you guys for convenience. I don’t like how Twitter is going about this – effectively creating a monopoly for consumption of Twitter data. It’s their data, so I suppose I can’t complain too much, but didn’t Microsoft get into a lot of trouble for making people use Internet Explorer all those years ago? How is this different in principle?

Share This