Taming Twitter with Microsoft Azure

Twitter is noisy.

At least 50% of tweets are duplicates, retweets or auto-generated from old content. And another 25% are likely tweets that, if you're busy, you don't really care to see at the moment. Thus, trying to use Twitter as a source for learning and new information is often time-consuming and not worth the effort. For example, on the topic of Dynamics 365/CRM and Azure, people and bots around the world send out over 1,000 tweets per day -- far too many for most of us to view.

Twitter is useful.

On the other hand, there are tweets that are important (or at least interesting) to see. It's a great resource for keeping up-to-date on just about any topic.

Twitter can be tamed.

This blog post demonstrates one way to tame Twitter: To cast aside unwanted or duplicate tweets to get to the good stuff.

The solution described below utilizes various Microsoft Azure and Office 365 services to achieve the following goals:

• Receive Tweets by e-mail that match a keyword or hashtag
• Eliminate duplicate tweets, retweets and bot tweets
• Eliminate tweets that contain a particular keyword (e.g., #jobs)
• Categorize tweets within e-mail folders
• Benefit from Twitter by significantly reducing the "noise"
• Have full control over the tweet filtering process to further enhance the solution

Get Ready to Build

If you're like me and find benefit in using Twitter but find it too time-consuming to wade through the noisy tweets, and if you're a DIY type of person who would rather build a solution then buy one, then this tweet tamer solution will be worth the effort.

Solution Components

This solution utilizes the following components:

• Microsoft Azure:
• 1 Logic App
• 1 Azure Function
• 1 Azure Storage Queue
• 2 Azure Storage Tables
• Microsoft Office 365
• 1+ Microsoft Flow workflows
• Outlook (desktop or web)

Time to implement: About 1 hour

Cost per month: It depends. If your Twitter searches are relatively narrow, running this solution might be free or not more than a few dollars per month. To avoid a big bill, avoid searching Twitter for keywords or hashtags (in your Flows) that will return hundreds or thousands of tweets per query.

Prerequisites

• Microsoft Azure Subscription
• Microsoft Office 365 account
• Optional: Microsoft Flow (Can also use a Logic App in place of Flow)
• Some software development or scripting experience will be useful but it's not mandatory
• Download and install Microsoft Azure Storage Explorer

Solution Runtime Overview

In a nutshell, this tweet-taming solution works in this way:

1. Microsoft Flow monitors Twitter for specific hashtags, keywords or phrases and posts matching tweets to an Azure Storage Queue.
2. Azure Logic App reads messages from the queue every 30 minutes, sends tweet messages to an Azure Function for analysis and storage.
3. The Azure Function determines whether the tweet is a duplicate or is an unwanted tweet by keyword. It does this by storing and matching tweet messages with Azure Storage Tables.
4. The Azure Logic App conditionally sends the tweet message to an email account. Office 365 Email rule moves tweets to specific folders based on keywords.

Note: In addition to Microsoft Flow, it's also possible to retrieve tweets with an Azure Logic App and other means (e.g., Twitter API). I chose to use Flow for this solution because it's easier to add additional tweet retrievers with Flow compared with creating separate Logic Apps to capture tweets. Plus, Flow is included with my Office 365 account, so using it does not use up my Azure Subscription credits.

Implementing the Solution

This section provides the steps for creating this tweet-taming solution.

Note: Since the procedures for creating and managing Azure services changes often, and since there are lots of online resources available, I am not providing complete step-by-step instructions. For example, when I say to create an Azure Storage account, I don't provide the specific steps for doing that, but I believe I've provided sufficient instructions to understand and build the solution. It just might take some research on your part to complete a step that you haven't done before.

Create an Azure Resource Group for this solution

In the Azure Portal, start by creating a Resource Group in Azure to assign to the services that you'll be creating in the steps below. Using a Resource Group helps organize Azure services that are part of the same solution.

Create the Azure Storage Queue and Tables

1. Create an Azure Storage account.
2. In Microsoft Azure Storage Explorer (or similarly capable tool), create a queue named "tweets" and two tables, one named "tweets" and the other named "unwantedTweetKeywords". There's no need to add additional columns to the Tables.

Create the Azure Function

1. Get the Azure Function files from the GitHub Gist location.
2. Create an Azure Function App. Name it something like "TweetTamer".
3. Create the Azure Function. Use the GenericWebHook-CSharp template. Name it "ProcessTweet".
4. Create the Azure Function files to correspond with the files you retrieved from the provided Gist location. Your function should have three files: run.csx, function.json and project.json.

Create the Azure Logic App

Create the Logic App as shown in the provided screenshots. Name it "ProcessTweets" or any name that you choose.

The "ProcessTweet" action shown below is where the Logic App calls the Azure Function. Construct the JSON string to pass to the function. If the return value from the function does not contain "ignore_tweet" then the tweet text is sent via email. The last step removes the Azure Storage queue message.

Create Rules In Outlook to Move Tweets

The solution is almost ready for use. One optional but recommended step is to create one or more email routing rules in your email application to move emails that start with "New Tweet" to a particular folder. Doing this will help keep your inbox clear of tweets and will help categorize the tweets to further put them into context.

Create Microsoft Flow workflows to receive Tweets and Send to Queue

The last step is to create one or more Microsoft Flow workflows to periodically query Twitter and send tweets to the Azure Storage queue that you created in a previous step.

1. Login to Microsoft Flow
2. Create a blank Flow.
3. Search for "Twitter" and select the trigger named "Twitter - When a new tweet is posted"
4. For the search text, enter one or more hashtags or keywords separate by the word "OR". Example: #msdyn365 OR #msdyncrm OR #dynamics365
5. Click "Next Step" and search actions for "azure queues". Select the "Azure Queues" action then select "Azure Queues - Put a message on a queue".
6. For the queue name, enter "tweets". For the message, select the "Tweet text" option from the dynamic content list. If you want to include a URL to the tweet, then see the image below for an example of how to construct the URL.
7. Click "Create flow" to save the flow and then click "Done".

Twitter is Tamed!

Within 15 minutes of creating one or more of the Microsoft Flow workflows that retrieve tweets and sends them to the Azure Storage queue, you should start receiving tweets in email.

Over time, the number of tweets you receive will decrease because you will not be receiving as many duplicate tweets, no retweets and no tweets that contain the keywords you specify (see "Ongoing Maintenance" below for details).

The end result is that you can now benefit from Twitter without the noise!

Here's my own tweet about this blog post in Outlook online...

Ongoing Maintenance

Ignoring Tweets by hashtag or keyword

As you receive tweets in email, when you see a word or hashtag appear that you'd like to ignore, you can add the word or hashtag to the Azure Storage table "unwantedTweetKeywords".  For the PartitionKey, enter "Keyword" and for the RowKey, enter the word or hashtag. Since Azure Storage tables don't allow the pound character ("#") in the RowKey column, use "HT" instead. For example, to ignore tweets that have #jobs in the text, enter "HTjobs" into the RowKey column. The Azure Function provided in this solution converts "HT" to the "#" character.

Other Ideas

Below are some other thoughts on using and extending this solution.

• Within your email client, create a folder named "Tweets to Read" and use it to store tweets (links) that you want to review later.
• Also within the email client, one idea is to create a folder for tweets that you want to share with others. You can set up a Microsoft Flow workflow to monitor a "To Share" folder and post the tweets on Slack or Yammer.