Seth Grimes, Author at Social Media Explorer

Interview With Jason Baldridge on Driving Social Marketing With Behavior Analytics

Seth Grimes — Tue, 05 Jul 2016 14:26:37 +0000

Computational linguists and computer scientists, among them University of Texas professor Jason Baldridge, have been working for over fifty years toward algorithmic understanding of human language. They’re not there yet. They are, however, doing a pretty good job with important tasks such as entity recognition, relation extraction, topic modeling, and summarization. These tasks are accomplished via natural language processing (NLP) technologies, implementing linguistic, statistical, and machine learning methods.

Computational linguist Jason Baldridge, co-founder and chief scientist of start-up People Pattern

NLP touches our daily lives, in many ways. Voice response and personal assistants — Siri, Google Now, Microsoft Cortana, Amazon Alexa — rely on NLP to interpret requests and formulate appropriate responses. Search and recommendation engines apply NLP, as do applications ranging from pharmaceutical drug discovery to national security counter-terrorism systems.

NLP, part of text and speech analytics solutions, is widely applied for market research, consumer insights, and customer experience management. The more consumer-facing systems know about people — individuals and groups — their profiles, preferences, habits, and needs — the more accurate, personalized, and timely their responses. That form of understanding — pulling clues from social postings, behaviors, and connections — is the business Jason’s company, People Pattern, is in.

I think all this is cool stuff so I asked two favors of Jason. #1 was to participate in a conference I organize, the up-coming Sentiment Analysis Symposium, taking place July 12 in New York. He agreed. (SME readers save 15% with the EXPLORER registration code.) #2 was to respond to a series of questions laid out below:

The Analytics of Language, Behavior, and Personality

Seth Grimes: People Pattern seeks to infer human characteristics via language and behavioral analyses, generating profiles that can be used to predict consumer responses. What are the most telling, the most revealing sorts of thing people say or do that, for business purposes, tells you who they are?

Jason Baldridge: People explicitly declare a portion of their interests in topics like sports, music, and politics in their bios and posts. This is part of their outward presentation of themselves: how they wish to be perceived by others and which content they believe will be of greatest interest to their audience. Other aspects are less immediately obvious, such as interests revealed through the social graph. This includes not just which accounts they follow, but the interests of the people they are most highly connected to (which may have been expressed in their posts and their own graph connections).

A person’s social activity can also reveal many other aspects, including demographics (e.g. gender, age, racial identity, location, and income) and psychographics (e.g. personality and status). Demographics are a core set of attributes used by most marketers. The ability to predict these (rather than using explicit declarations or surveys) enables many standard market research questions to be answered quickly and at a scale previously unattainable.

SG: And what can one learn from these analyses?

People Pattern portrait search

Jason: Personas and associated language use.

As a whole, this kind of analysis allows us to standardize large populations (e.g. millions of people) on a common set of demographic variables and interests (possibly derived from people speaking different languages), and then support exploratory data analysis via unsupervised learning algorithms. For example, we use sparse factor analysis to find the correlated interests in an audience and furthermore group the individuals who are best fits for those factors. We call these discovered personas because they reveal clusters of individuals with related interests that distinguish them from other groups in the audience, and they have associated aggregate demographics—the usual things that go into building a persona segment by hand.

We can then show the words, phrases, entities, and accounts that the individuals in each persona discuss with respect to each of the interests. For example, one segment might discuss Christian themes with respect to religion, while others might discuss Muslim or New Age ones. Marketers can then use these to create tailored content for ads that are delivered directly to the individuals in a given persona, using our audience dashboard. There are of course other uses, such as social science questions. I’ve personally used it to look into audiences related to Black Lives Matter and understand how different groups of people talk about politics.

Our audience dashboard is backed by Elastic Search, so you can also use search terms to find segments via self-declared allegiances for such polarizing topics.

A shout-out —

Personality and status are generally revealed through subtle linguistic indicators that my University of Texas Austin colleague James Pennebaker has studied for the past three decades and is now commercializing with his start-up company Receptiviti. These include detecting and counting different types of words, such as function words (e.g. determiners and prepositions) or cognitive terms (such as “because” and “therefore”), and seeing how a given individual’s rates of use of those word classes compares to known profiles of the different personality types.

SG: So personas, language use, topics. How do behavioral analyses contribute to overall understanding?

Jason: Many behaviors reveal important aspects about an account that a human would struggle to infer. For example, the times at which an account regularly posts is a strong indicator of whether they are a person, organization or spam account. Organization accounts often automate their sharing, and they tend to post at regular intervals or common times, usually on the hour or half hour. Spam accounts often post at a regular frequency — perhaps every 8 minutes, plus or minus one minute. An actual person posts in accordance with sleep, work, and play activities, with greater variance — including sporadic bursts of activity and long periods of inactivity.

SG: Any other elements?

Jason: Graph connections are especially useful for bespoke, super-specific interests and questions. For example, we used graph connections to build a pro-life/pro-choice classifier for one client to rank over 200,000 individuals in greater Texas on a scale from most likely to be pro-life to most-likely to be pro-choice. By using known pro-life and pro-choice accounts, it was straightforward to gather examples of individuals with a strong affiliation to one side or the other and learn a classifier based on their graph connections that was then applied to the graph connections of individuals who follow none of those accounts.

SG: Could you say a bit about how People Pattern identifies salient data and makes sense of it, the algorithms?

Jason: The starting point is to identify an audience. Often this is simply the people who follow a brand and/or its competitors, or who comment on their products or use certain hashtags. We can also connect the individuals in a CRM to their corresponding social accounts. This process, which we refer to as stitching, uses identity resolution algorithms that make predictions based on names, locations, email addresses and how well they match corresponding fields in the social profiles. After identifying high confidence matches, we can then append their profile analysis to their CRM data. This can inform an email campaign, or be the start for lead generation, and more.

Making sense of data — let’s look at three aspects — demographics, interests, and location —

Our demographics classifiers are based on supervised training from millions of annotated examples. We use logistic regression for attributes like gender, race, and account type. For age, we use linear regression techniques that allow us characterize the model’s confidence in its predictions — this allows us to provide more accurate aggregate estimates for arbitrary sets of social profiles. This is especially important for alcohol brands that need to ensure they are engaging with age-appropriate audiences. All of these classifiers are backed by rules that detect self-declared information when it is available (e.g. many people state their age in their bio).

We capture explicit interests with text classifiers. We use a proprietary semi-supervised algorithm for building classifiers from small amounts of human supervision and large amounts of unlabeled texts. Importantly, this allows us to support new languages quickly and at lower cost, compared to fully supervised models. We can also use classifiers built this way to generate features for other tasks. For example, we are able to learn classifiers that identify language associated with people of different age groups, and this produces an array of features used by our age classifiers. They are also great inputs for deep learning for NLP and they are different from the usual unsupervised word vectors people commonly use.

For location, we use our internally developed adaptation of spatial label propagation. With this technique, you start with a set of accounts that have explicitly declared their location (in their bio or through geo tags), and then these locations are spread through graph connections to infer locations for accounts that have not stated their location explicitly. This method can resolve over half of individuals to within 10 kilometers of their true location. Determining this information is important for many marketing questions (e.g. how does my audience in Dallas differ from my audience in Seattle?) It obviously also brings up privacy concerns. We use these determinations for aggregate analyses but don’t show them at the individual profile level. However, people should be aware that variations of these algorithms are published and there are open source implementations, so leaving their location field blank is by no means sufficient to ensure your home location isn’t discoverable by others.

SG: My impression is that People Pattern, with an interplay of multiple algorithms and data types and multi-stage analysis processes, is a level more complex than most new-to-the-market systems. How do you excel while avoiding over-engineering that leads to a brittle solution?

Jason: It’s an ongoing process, with plenty of bumps and bruises along the way. I’m very fortunate that my co-founder, Ken Cho, has deep experience in enterprise social media applications. Ken co-founded Spredfast [an enterprise social media marketing platform]. He has strong intuitions on what kind of data will be useful to marketers, and we work together to figure out whether it is possible to extract and/or predict the data.

We’ve struck on a number of things that work really well, such as predicting core demographics and interests and doing clustering based on those. Other things have worked well, but didn’t provide enough value or were too confusing to users. For example, we used to support both interest-level keyword analysis (which words does this audience use with respect to “music”) and topic modeling, which produces clusters of semantically related words given all the posts by people in the audience, in (almost) real-time. The topics were interesting because they showed groupings of interests that weren’t captured by our interest hierarchy (such as music events), but it was expensive to support topic model analysis given our RESTful architecture and we chose to deprecate that capability. We have since reworked our infrastructure so that we can support some of those analyses in batch (rather than streaming) mode for deeper audience analyses. This is also important for supporting multiple influence scores computed with respect to a fixed audience rather than generic overall influence scores.

Ultimately, I’ve learned to think about approaching a new kind of analysis not just with respect to the modeling, but as importantly to consider whether we can get the data needed at the time that the user wants the analysis, how costly the infrastructure to support it will be, and how valuable it is likely to be. We’ve done some post-hoc reconsiderations along these lines, which has led to streamlining capabilities.

SG: Other factors?

Jason: Another key part of this is having the right engineering team to plan and implement the necessary infrastructure. Steve Blackmon joined us a year ago, and his deep experience in big data and machine learning problems has allowed us to build our people database in a scalable, repeatable manner. This means we now have 200+ million profiles that have demographics, interests and more already pre-computed. More importantly, we now have recipes and infrastructure for developing further classifiers and analyses. This allows us to get them into our product more quickly. Another important recent hire was our product manager Omid Sedaghatian. Omid is doing a fantastic job of figuring out what aspects of our application are excelling, which aren’t delivering expected value, and how we can streamline and simplify everything we do.

SG: Excuse the flattery, but it’s clear your enthusiasm and your willingness to share your knowledge are huge assets for People Pattern. Not coincidentally, your other job is teaching. Regarding teaching — to conclude this interview — I recruited you to speak at the 2016 Sentiment Analysis Symposium in New York, and pre-conference you’ll present a tutorial, Computing Sentiment, Emotion, and Personality. Could you give us the gist of the material you’ll be covering?

Jason: Actually, I just did. Well, almost.

I’ll start the tutorial with a natural language processing overview and then cover sentiment analysis basics — rules, annotation, machine learning, and evaluation. Then I’ll get into author modeling, which seeks to understand demographic and psychographic attributes based on what someone says and how they say it. This is in the tutorial description: We’ll look at additional information that might be determined from non-explicit components of linguistic expression, as well as non-textual aspects of the input, such as geography, social networks, and images, things I’ve described in this interview. But with an extended, live session you get depth and interaction, and an opportunity to explore.

Thanks Jason. I’m looking forward to your session.

The post Interview With Jason Baldridge on Driving Social Marketing With Behavior Analytics appeared first on Social Media Explorer.

Counting Bugs at LinkedIn

Seth Grimes — Tue, 27 Jan 2015 11:00:15 +0000

LinkedIn has a bug problem, in two senses. There are long-standing, unresolved errors, and there are agitators like me (or is it only me?) who keep finding more and say so.

This article is my latest “bug LinkedIn” entry. My latest finds center on counting. They’re very visible. I’ll show you two instances, and toss in a screenshot of a special slip-up.

(See also My Search for Relevance on LinkedIn, on search-relevance deficiencies, posted in March 2014; my April 2014 via-Twitter reporting of incorrect rendering of HTML character entities, here and here, since fixed although a very similar error in LinkedIn e-mail notifications remains unresolved; a February 2013 Twitter thread about the lack of needed LinkedIn profile spell-checking — do a LinkedIn search on “analtics” (missing Y) and you’ll see what I mean; and my July 2012 LinkedIn, Please Take on Group Spammers, still very much an issue. And the flawed algorithm I reported in my July 2014 article, LinkedIn Misconnects Show That Automated Matching is Hard, remains uncorrected.)

So what’s new?

How Many Moderation Items?

LinkedIn groups are a great feature. I belong to quite a few, and I moderate several.

Check out the moderation screen to the right. The Discussions tab is displayed, indicating 35 items in the Submissions Queue pending approval — the same number is shown next to the Manage menu-bar item — except we don’t see any pending discussion submissions, do we? We do, however, see a tick box next to… nothing.

Counting bug #1.

Actually, I can explain this error. LinkedIn provides group moderators a Block & Delete option under the Change Permissions drop down, as shown in the right-middle of my shot. A diligent moderator will use it to ban group members who repeatedly submit off-topic content. I’ve used Block & Delete. Each time I use it, while the submitted items disappear, they’re still being counted. My guess is that they’re still in LinkedIn’s database, but now flagged with a “blocked” status.

So we have a bad counting query that can be easily fixed. All LinkedIn Engineering has to do is add a condition — if in SQL, in the WHERE clause — so that only non-blocked moderation entries are counted.

How Many Connection Requests?

Counting error #2 involves connections requests. It’s a two-fer — two errors, actually. I can’t explain them, but I can show them and describe them.

First, check out the Inbox image, which shows a connection invitation that I’ve already accepted. Note the “1st” next to the person’s name. The second image confirms that he and I are connected. Look closely at his profile and you will see “Connected 1 day ago.”

The second image also the drop-down under “Add Connections,” which, again erroneously, shows a pending connection invitation from the person I’m already connected to.

But that’s not all! Did you notice the little plus-person icon in the upper-right of each of those screens? Did you notice the number displayed in a red background? It’s 3. Now, how many connection requests do you see under Invitations in the drop-down of the second image. I see 2.

Counting error #2.

Placeholder

And finally, a lagniappe, an extra for those who have read this far. Check out the item under “Ads You May Be Interested In” in the image to the right

Placeholder?

A Loyal, Paying User

Finally, let me reassert that I am a loyal, paying LinkedIn user. Did you notice the word “premium” next to the “in” logo in the screenshots I posted?

There’s always room for improvement, and of course, LinkedIn capabilities have advanced light years since I wrote, in InformationWeek in 2004, “LinkedIn is the only social-networking system I looked at that currently deserves enterprise consideration.” Myself, I may be a more astute industry analyst and better writer now, in 2015, than I was then. Here’s to progress!! … and also to getting even the little things right.

The post Counting Bugs at LinkedIn appeared first on Social Media Explorer.

A Cheap Way to Discover Brand-Topic Affinities on Twitter

Seth Grimes — Mon, 11 Aug 2014 10:00:58 +0000

Sometimes interesting things appear when you’re not even looking. And some lessons taught are applicable far beyond immediate challenges.

Case in point: The realization that Twitter advertising statistics can reveal brand-topic affinities.

Ad stats help you assess how well you’ve targeted promoted tweets — that’s their purpose — but you can use them for much more. You can use them to study competitive positioning and identify influencers around particular topics of interest. The trick is to craft tweets that don’t (only) promote a product or service, but also/instead help you evaluate the topic-engagement link. The insights revealed aren’t especially useful for me — I’m well-positioned in my text and sentiment analysis consulting specialization — but if your business depends on precision online targeting, you may find ads data to be a new, unique, inexpensive source of social intelligence.

Insights from Twitter Engagement

I’ll save you a long read: I ran two Twitter promoted tweet campaigns. One targeted a set of keywords. For the second, I entered a set of @handles to target people similar to those accounts’ followers. I promoted a single tweet, one associated with a well defined technology topic.

The targeted @handles: Each represents a brand, whether an organization, product, or person. IBM is a brand, and so are @IBMWatson, IBM Watson evangelist Fredrik @Tunvall, and Gartner analyst @Doug_Laney, whose coverage extends to Watson.

What I advertised isn’t important beyond that the ad content was single-topic and brand-neutral. Brand-neutrality reduces response bias, whether toward or against a brand. The single-topic focus eliminates ambiguity; it makes clear what prompted the response. Net is that we can associate engagement — retweets, replies, follows, and other clicks — with one, particular topic. Ad stats break out and rank engagement by targeted @handle and by keyword, giving us neat way to study affinities.

My @handle-targeted campaign achieved a 5.96% engagement rate, which I consider pretty good. Twelve of the @handles I targeted had over 10% engagement, out of 69 with at least 100 ad impressions, and seven were below 4%.

We learn from the variation, from the spread of response rates. We learn which brands are associated with a topic and which aren’t. The @handles for individuals: High engagement rates reveal or confirm influencers for the tweet’s topic. The uses of company and product @handle-topic associations is close to self-evident so I won’t elaborate on them.

Get a complete set of insights by running a parametric study, a series of ads with topics whose associations you wish to explore, for a fixed set of target @handles. You may find surprises. I did. In the end, you’ll gain solid, valuable social intelligence.

Finally: Cheap. Twitter Ads per-engagement costs are very, very reasonable, and because you pay by the engagement rather than by the impression, you’re not penalized for poorly composed or targeted advertising. (But please, let’s not waste anyone’s time.) I won’t tell you what I spent on my Twitter advertising, just that the amount was modest, with excellent return on investment.

More Detail

My promoted tweet advertised a free report I recently published, delivering findings from my Text Analytics 2014 market study. The term “text analytics” describes a collection of technologies and processes that extract information from social, online, and enterprise text. My advertising aim was click-throughs to the download page, and secondarily, retweets and other forms of Twitter engagement.

(Twitter does offer additional advertising options, for instance lead generation cards and conversion tracking, useful for ad optimization but not for the affinity study I’m describing in this article.)

I chose to target 77 Twitter @handles, of solution providers that sell text analytics products or services, of industry analysts who cover text analytics or application areas, and of associations. Text analytics is commonly applied in customer experience management, market research, social intelligence, financial services, media and publishing, and public policy, so I included certain companies, analysts, and consultants who work in those domains.

(An ideal way to learn more is to check out a conference I’m organizing, LT-Accelerate, slated for December 4-5, 2014 in Brussels.)

As I’ve mentioned, results — ad engagement — varied widely.

Top scorer was @Confirmit, a survey research/insights firm, at 15.62%. Two in every thirteen promoted-tweet impressions led to a click, favorite, or retweet. I think that’s pretty good.

In the cellar, @The_ARF (the Advertising Research Foundation) at 0.57%.

The easy conclusion is that Confirmit and other top-scorers — @GateAcUk (GATE open-source text analytics) and @SAPAnalytics — have strong text analytics brand interest while only a small portion of the ARF’s audience has a text analytics affinity. SAP and Confirmit will want to play to the first point, while frankly, I may put less personal effort into working with the ARF.

I’ll paste in my full set of results below.

Complications

Finally, I’d be remiss if I didn’t discuss complications.

Secondary data use — analysis of data that was collected and reported for purposes other than your current ones — is rarely straightforward. The available data may not fit your preferred categories or characteristics — for instance, you might want hourly data, but daily is the best you can get — or you might be not have access to detailed metadata that fully describes the data and collection conditions.

There is a lot of follower overlap among the @handles I targeted. While I could cross-check follower lists, combinatorics suggest an intractable attribution task. If you need to account for ad engagements across a set of @handles (or ad-targeting keywords), I suggest running simultaneous, separate ad campaigns, one for each @handle, or choose yet another option, the one I chose: Don’t overthink your experiment, because you most likely don’t need highly precise results.

Results

The following are my promoted-tweet campaign engagement results, for @handles with at least 75 impressions:

@handle	Impressions	Engagements	Rate
Campaign totals	21,335	1,271	5.96%
@confirmit	160	25	15.62%
@GateAcUk	229	32	13.97%
@SAPAnalytics	115	16	13.91%
@LTInnovate	90	12	13.33%
@metabrown312	256	29	11.33%
@allegiancetweet	135	15	11.11%
@texifter	310	34	10.97%
@DataSift	105	11	10.48%
@SASsoftware	621	65	10.47%
@havasi	374	39	10.43%
@Verint	204	21	10.29%
@dreasoning	605	62	10.25%
@jasonbaldridge	130	13	10.00%
@Lexalytics	196	18	9.18%
@Clarabridge	1265	116	9.17%
@lousylinguist	178	16	8.99%
@bobehayes	314	28	8.92%
@DeloitteBA	901	80	8.88%
@sinequa	102	9	8.82%
@LuminosoInsight	616	54	8.77%
@eMarketer	165	14	8.48%
@digimindci	489	41	8.38%
@Gartner_inc	621	52	8.37%
@basistechnology	798	66	8.27%
@IDC	289	23	7.96%
@Medallia	384	30	7.81%
@RapidMiner	1,598	123	7.70%
@NetBase	326	25	7.67%
@Expert_System	1,125	86	7.64%
@stuartrobinson	119	9	7.56%
@NICE_Systems	292	22	7.53%
@dtunkelang	859	64	7.45%
@nik	997	74	7.42%
@Doug_Laney	178	13	7.30%
@ClickZ	207	15	7.25%
@kdnuggets	318	23	7.23%
@IBMWatson	83	6	7.23%
@40deuce	629	45	7.15%
@TEMIS_Group	261	18	6.90%
@btemkin	588	40	6.80%
@strataconf	240	16	6.67%
@forrester	967	62	6.41%
@adage	360	23	6.39%
@Brandwatch	454	29	6.39%
@stanfordnlp	580	37	6.38%
@Bazaarvoice	795	49	6.16%
@rwang0	440	27	6.14%
@attensity	718	44	6.13%
@LoveStats	631	38	6.02%
@CXPA_Assoc	389	23	5.91%
@Smartlogic_com	871	51	5.86%
@LithiumTech	1,321	75	5.68%
@visible	881	49	5.56%
@crimsonhexagon	1,415	78	5.51%
@Synthesio	758	40	5.28%
@Econsultancy	438	23	5.25%
@pgreenbe	305	16	5.25%
@HPAutonomy	422	21	4.98%
@RecordedFuture	487	22	4.52%
@Gnip	45	2	4.44%
@IBMAnalytics	91	4	4.40%
@coveo	989	43	4.35%
@JeanneBliss	625	27	4.32%
@TomHCAnderson	510	21	4.12%
@digimind_FR	258	9	3.49%
@ekolsky	378	12	3.17%
@KISSmetrics	2,869	86	3.00%
@etuma360	139	4	2.88%
@comScore	2,972	80	2.69%
@converseon	271	7	2.58%
@The_ARF	1,585	9	0.57%

The post A Cheap Way to Discover Brand-Topic Affinities on Twitter appeared first on Social Media Explorer.

LinkedIn Misconnects Show that Automated Matching is Hard

Seth Grimes — Wed, 09 Jul 2014 10:00:23 +0000

I’ll report here on a LinkedIn error — it’s not a bug, it’s a flawed algorithm, significant although far from earth-shattering — that shows how difficult automated matching can be. I’ll then offer practical steps LinkedIn could take toward accurate matching.

Why should you read on? Not only because (I’m guessing) you have a LinkedIn profile, but also because in an “omni-channel” world, data matching — also known as data integration, data fusion, record linkage, and synthesis — is central to meeting everyday social and enterprise business challenges.

Two Mismatches

Check out the snippet to the right, from my own hand-entered LinkedIn profile. You’ll see that I wrote for Intelligent Enterprise magazine for quite few years. (Jeannette Boyne, thanks again for the recommendation!)

But what’s with the Intel Corporation information that appears when I hover over “Intelligent Enterprise magazine (CMP)”? It’s a LinkedIn-generated misconnection. Intelligent Enterprise magazine was folded into InformationWeek a few years back. IE ceased to exist as a free-standing brand. LinkedIn must have taken the first five characters of the magazine’s name and decided that Intel is the best match.

This matching error is significant because for LinkedIn, a connections company, social-graph accuracy is gold. Members hand-craft their networks based on past, current, and hoped-for future business relationships. LinkedIn derives People You May Know recommendations from our employment histories and interests, but recommendations are only suggestions. We see that when LinkedIn asserts connections, as the platform does in the example I show, the company gets into trouble.

(LinkedIn also misses certain computable connections, more on which later.)

LinkedIn actually mismatches a second of my former employers, Magnet Interactive. I didn’t work for the company you’ll see in the mouseover pop-up when you visit my profile, or for any company related to it. Just because two company names look the same, doesn’t mean they’re the same company!

So we have two examples of “entity resolution” false positives in just my profile. (LinkedIn should pay me a product-quality bounty. See also My Search for Relevance on LinkedIn, posted in March; my April via-Twitter reporting of incorrect rendering of HTML character entities, here and here; a February 2013 Twitter thread about the lack of needed LinkedIn profile spell-checking; and my July 2012 LinkedIn, Please Take on Group Spammers.)

I found additional examples by looking at profiles of others who were formerly employed by now-defunct companies. Here’s one such example, in the image to the right. Hyperion Software was acquired by Arbor Software — as this person’s profile states! — which in turn was acquired by IBM.

Funny thing: Somehow, as you can see toward the top of the image, LinkedIn did get right that PeopleSoft was acquired by Oracle.

Automated Matching is Hard

Yup, automated matching is hard. Direct marketers and others have been working the problem for years, for instance, in order to merge and deduplicate mailing lists. Software tools may declare record matches when the values of several fields in pairs of records line up — for instance, first initial + last name + address — and they may tolerate abbreviations, misspellings, and data variations (PA = Penna. = Pennsylvania = Pennslvania) or even exploit phonetic similarity in names. Some even determine that fields in different databases have the same meaning, despite different field names, based on data profiling, based on a scans of the fields’ values. The matches are sometimes fuzzy, decided based on a probability judgment.

Check out database-systems wizard Mike Stonebraker’s latest, Tamr, which aims to overcome the data disconnect.

FirstRain, which extracts, aggregates, and organizes business information from online and social sources, provides an even better example of semantic matching done right. As described in words pulled from FirstRain’s Web site: “Selling to GE Locomotives? You won’t want to read a generic newsfeed on GE Aviation or GE Capital — and with FirstRain, you won’t. You will only see what’s relevant based on how you sell and market to each specific business line within a company.” That is, the company (claims it) has successfully addressed the semantic-matching problem, at a level of granularity, the division level, that exceeds the LinkedIn matching need’s.

I wrote about FirstRain and a number of other semantic-matching successes — Tableau, Attivio, Google, and the now-defunct Extractiv — in my 2011 InformationWeek article, 5 Paths To The New Data Integration. (I’m linking you to page 2, which features Attivio and FirstRain.)

A Company Graph

My prescription for LinkedIn: Create a company graph, an ontology that recognizes factors that include:

naming variations (e.g., General Motors = General Motors Corporation = (sometimes) GM);

hierarchy (Chevrolet is a GM division);

temporality and geography (Digital Equipment Corporation was founded in Massachusetts in 1957 and existed under that name through 1998, with a first international office opening in West Germany (now just Germany, of course) in 1963);

transactions (Arbor Software merged with Hyperion Software, formerly IMRS, to form Hyperion Solutions Corporation, which was in turn acquired by IBM);

multiple uses of a single name (polysemy) (SAS is both an enterprise software company and an airline); and

identity shifts (SAS, as in the software company, once stood for Statistical Analysis System, and the company was called SAS Institute; SAS, the airline, was once Scandinavian Airlines System).

LinkedIn has the data to create just such a company graph, via text mining, and surely employees have the data-science smarts. How-to examples? Two:

That profile pictured above that lists Hyperion Software as an employer: It explicitly states, “Company acquired by Arbor Software.” Text analytics will identify “Company” as a contextual anaphoric reference to Hyperion Software. (Sorry about the jargon. Pronouns such as he and she are other commonly found anaphora.) Text analytics will identify Arbor Software as a named entity and will discern and extract the “acquired by” relationship. Further, I’d bet there are other LinkedIn profiles that corroborate this particular corporate acquisition.

Now refer back to my first image, above. Jeannette Boyne, who recommended me, lists in her profile that she worked as “Senior Editor, Intelligent Enterprise” for “CMP Media (div of United Business Media) from September 1998 to September 2005, overlapping the years I list for my “Intelligent Enterprise magazine (CMP)” association. Our profiles, and profiles of other former associates with whom we’re both linked, provide data that supports high-confidence entity (company name) and subsidiary-relationship resolution. LinkedIn, lacking sufficient semantic smarts, despite Jeannette’s and my first-degree connection and her recommendation and the similarity of the employer names, failed to infer an obvious connection.

The Unreasonable Effectiveness of Data… and Analytics

Consider this column a call to action, for LinkedIn and for other data-rich, capable organizations.

Do you want to lead in digital? Simplistic approaches — Assuming associations based only on a match in the first few characters of two names!? — don’t cut it. Use your data. Create and apply knowledge structures — graphs, ontologies, semantic networks — to resolve and disambiguate names and extract relationships. Apply multiple methods, performing cross-checks until you’re reasonably certain about inferences.

LinkedIn, if you’re not going to take these steps: Better to provide no results — skip the possibly mis-inferred connections — rather than erroneous ones. But consider that high-quality data, and high-value results, are worth the extra effort. Users will thank you.

The post LinkedIn Misconnects Show that Automated Matching is Hard appeared first on Social Media Explorer.

My Search for Relevance on LinkedIn

Seth Grimes — Fri, 21 Mar 2014 10:00:31 +0000

I’m a heavy LinkedIn user, and like many of my ilk — recruiters, marketers, job seekers — I’m quick to jump to search for people-finding. That’s because exploring a social network, connection-by-connection, is beyond tedious. All but poorly-connected users will have millions in their networks, within 2 or 3 links, and manual exploration simply doesn’t scale. (Cutting connections isn’t the answer; it destroys a network’s value, which grows steeply with the number of participants according to the network effect.) Further, you can’t explore what you can’t see. Because many individuals hide their connection lists, a connection-surfer won’t see entire branches of her network. So I — and you — use search.

Unfortunately, for all the professional social platform’s data-science mojo, elements of LinkedIn search are disappointingly weak. I’ll illustrate those elements and describe how LinkedIn can better help users discover business value in their connections.

The State of Search on LinkedIn

The search user interface gold standard is a simple text-entry box, and LinkedIn obliges, with the added possibility of using a drop-down to focus search on People, Jobs, Companies, Groups, Universities, Articles, and Inbox. (That latter label is strange, given that “Inbox” search searches all your messages. And by the way, I’m using the Web interface although my points apply even more to the mobile app, which, as is typical, offers reduced capabilities.) Or click on Advanced and get this window:

And here we see the weakness. LinkedIn offers People Search, fine if you’re looking for individuals, identified by particular attributes, by particular facet values. LinkedIn does not offer search that exploits its essential identity; LinkedIn does not offer Network Search.

Your Networks, Via Your Connections

I’ll explain what I mean by “network search” by showing that the data is there, in LinkedIn. Consider a LinkedIn search results screen, mine:

(Actually, this screen represents a null search; it’s the start of a listing of my connections.)

We have faceted navigation — the ability to filter on values of characteristics including distance of connection (1st, 2nd, Group) and physical location — and we can see network attributes, in particular, the number of connections I share with each individual.

I don’t have a clue how LinkedIn did the ranking here, despite the (fairly opaque) explanation the company provides. For instance, of the seven individuals you see displayed, I’ve met only two in person, and only one of them more than once, Howard Dresner. Howard and I share 138 connections; that our networks so significantly overlap is a strong indicator of profile similarity, his and mine. Yet the top-of-the-list individual and I share only eight connections, I’ve never met him in person, and I’ve interacted with his company on only one occasion. How did he get to be top ranked for me?

We’re in a damned-if-you-do, damned-if-you-don’t situation. If the answer to this apparently anomalous ranking is that it’s an unordered list, not a ranking, then the question is, Why not? And if the answer is that LinkedIn is applying a ranking algorithm that makes sense only to the company’s engineers but not to a user like me, then the questions are, WTF? and Why can’t I have the ranking my way?

I want to be able to order hits by descending number of shared connections. I can’t, and that’s because, again, LinkedIn offers only People Search when I also want Network Search.

It’s a Network, Not Just a Set of Nodes

LinkedIn hosts social networks. In social networks, the number of short connections between two individuals in the graph is a good indicator of similarity, of shared interests. But I don’t want only to rank search hits by number of connections. I want to use other interesting network properties in my searches. LinkedIn gets the concepts; they just don’t let me use them. What concepts? Clusters, or subnets if you will. Take a look at my LinkedIn InMap:

Each colored node-set represents individuals who share a certain affinity with me, based on network structure. LinkedIn says, “We use information about how people in your network are connected to you and each other to create your personalized map. Groups like colleagues, people you went at school with, or friends are separated into color-coded clusters, as people within these groups are also interconnected with each other.”

Network Search would allow me to limit searches to members of particular clusters and their connections. In my case, that would mean I could restrict search to my BI & data warehousing connections (the azure cluster toward the top of my InMap) or my market research connections (pea green toward the lower right) or my European information retrieval and text analytics connections (reddish, lower left), in order to improve results relevance. What I’m asking is for LinkedIn to support a new, dynamic search facet.

Relevance is key, and leads me to my final topic, recommendations, which I’d characterize as a passive form of search.

When Will They Ever Learn?

I won’t accept a connection request from a recruiter or sales person whom I haven’t actually interacted with, nor from a student who doesn’t have any non-academic accomplishments that are professionally relevant to me, nor a request from a non-professional contact such as a community member. I follow this rule because I use LinkedIn almost exclusively for professional networking, and mixing in these (literal) outliers would make it much more difficult for me to correctly target my outreach efforts.

So who does LinkedIn recommend, as People You May Know? Check ’em out, in the image at the right. Nothing personal, but they are not people I will be connecting to, and LinkedIn should have known that.

One of them works professionally in an area related to mine, although not closely. A second shares one connection with me and lives near my community, with no professional connection however, and a third is three links away from me with our only similarity being that she also lives near my community.

I do, occasionally take LinkedIn connection recommendations, but never for people like these, and I have ignored many connection requests from people similar to them. I have done many, many LinkedIn searches, on terms such as “sentiment analysis” and “customer experience,” but never on terms that would turn up these people.

If LinkedIn did a bit of behavior mining — my search, profile viewing, and connection requests initiated or accepted — the platform’s People You May Know recommendations would be far more accurate. Machine learning could do the job; not a small task given the number and diversity of LinkedIn users, but then you don’t have to mine every user’s actions in order to come up with predictive recommendation models that would surely out-perform what’s in place now.

My Search for Relevance

I have high expectations for LinkedIn. I know that the company has great data-mining and information-retrieval capabilities, but also I pay them quite a bit. (I’ve been a Premium user on and off; I downgrade, to save money, for stretches when I’m not going to be using the platform extensively.)

So consider this article a challenge from a faithful user. Help me do my job better. Factor network characteristics and user behaviors into searches, and help me in my search for relevance.

The post My Search for Relevance on LinkedIn appeared first on Social Media Explorer.

Social Sentiment’s Missing Measures

Seth Grimes — Wed, 29 Jan 2014 11:00:53 +0000

Social-analytics accuracy is essential, whether you seek broad understanding of attitudes that affect your business, early warning of emerging threats, or to spot individual issues for customer care. Yet as I wrote in 2012, “focus on accuracy distracts users from the real goal, not 95% analysis accuracy but rather support for the most effective possible business decision making.” The most accurate and sophisticated listening/response program will fail if you’re not measuring values that matter and communicating useful and usable insights.

“Emotions tend to signal ‘what really matters’,” according to Rosalind Picard, professor at the MIT Media Lab. Given emotions’ importance, sentiment analysis is key to effective social intelligence. It’s also one of my focus areas: I organize a conference on sentiment and other “human data,” the Sentiment Analysis Symposium, coming up March 5-6, New York. I will concentrate on sentiment in this article, aiming to bring out principles that apply to all of social analytics. Let’s take a look then at social sentiment’s missing measures, and at other, metrics-related steps we can take to improve our business decision making.

Over-Simplification

Over-simplification is issue #1. I find that business analysts are too-often satisfied with crude positive/negative sentiment scoring. We can do better. There are two issues here. First issue is the over-simplification involved in shoe-horning sentiment into two catch-all categories. Second is the practice of scoring itself. Scoring is useful but reductionist; if you stop with a score, you will fail to see indicators and explanations that lie beyond. Add to these a third, linked point: If you treat analysis dashboards as the measurement end-product, you miss out on truly valuable insights, that can be gained by studying abstract attributes such as engagement, advocacy, and connection. And a fourth point: For true social intelligence, you need to reach beyond the metrics and indicators to explore root causes.

Evaluating each of these points —

I define sentiment analysis as a systematic attempt to identify, quantify, and exploit attitudes, opinion, and emotion in online, social, and enterprise sources. Attitudes, opinion, and emotion: These reflect everyday “affective” states; they convey our feelings about a person, product, brand, or policy.

We have the computing power and the data to classify sentiment in fashions that make business sense. Missing measure #1 is sentiment classified beyond positive/negative polarity, categorized instead according to business-aligned splits: promoter/detractor, satisfied/disappointed, happy/sad/angry, or whatever means business for you. Flexible automated methods, for instance supervised and unsupervised machine learning (that is, with and without predefined categories), but also expert or crowd-sourced human analysis, turn these analyses into just another classification problem. So don’t accept categories that aren’t the best match for your business problem.

Who gets this point? Vendors Kanjoya and Crimson Hexagon for two, but they, and other vendors I’ll cite in this article, are exceptions to the rule.

Scoring Points

Both an NPS and a sentiment score should be considered a starting point for deeper exploration

Promoter/detractor, which I suggested as a sentiment category pair, is famously measured via the question, “How likely is it that you would recommend Company X to a friend or colleague?,” with answers aggregated into a Net Promoter Score (NPS). A sentiment score is similarly a summation of positives and negatives, although in this case, the ratings are extracted via natural-language processing (NLP) from documents or messages or verbatim (free-text) survey responses. Both an NPS and a sentiment score should be considered a starting point for deeper exploration. On their own — and unlike a consumer-finance credit score — neither score is adequate to support decision making.

Blame the gap on the qualitative, subjective, and highly variable nature of the sentiment source material, typically text. Text-derived scores are unreliable for comparison purposes, carry no explanatory power, and mask other important measures… more on which toward the end of this article. For now, what measures lurk behind sentiment scores?

Behind the Scores

Missing measure #2 is sentiment density. When a lot of feeling is packed into a short text, that’s a message, review, or article your customer care, market research, or product quality staff should home in on. NetBase, via the Brand Passion Index, is an example of a vendor that shows it gets this need.

The challenge is that in mixed messages, positives and negatives may balance out to a near-neutral score. An example along the lines “The location was great, but I’ve never experienced worse service” — net mildly negative — illustrates why we need sentiment density as a new, distinct measure, to complement sentiment scores. A density measure could also help create comparability across sources, between 140-character-max tweets and long-form product reviews.

Refining sentiment resolution by creating scores for individual features or aspects, for instance for food versus service versus price as reported in restaurant reviews, will help, but you can still have a cancellation effect within each score. How can you surface a cancellation problem? By looking at missing measure #3, variation. A variation (or dispersion) metric would flag a mixed-ratings example like my location/service case. It would tell you to create more-granular scores (e.g., separate scores for service and price) or, if the tool you’re using isn’t capable of that refinement, it would flag cases for closer human examination.

To flag volatility, as an indicator of both risk and opportunity, we need a new measure

Similar to variation/dispersion is volatility, missing metric #4, a measure of variation over time. A basic trend line will help you see volatility, if the reporting is at the right frequency. If it’s not, the volatility — significant, rapid swings in mood or feeling — will be hidden. To flag volatility, as an indicator of both risk and opportunity, we need a new measure. Among vendors, the closest I’ve found to supporting these metrics is Social Market Analytics, which targets financial-market analyses.

Now, how else can we improve our social intelligence, and our overall business decision making?

Too Much Information, and Too Little

“I wouldn’t ask Facebook for another measurement. I’d ask it to cull the 95% of metrics that mean absolutely nothing to most social media marketers, let alone clients.”

There is such a thing as too much information. I was caught by one particular response to a blogged eConsultancy question, If you could ask Facebook for one new metric, what would it be? The reply from Peter Wood, UK social media director at STEAK, was “I wouldn’t ask Facebook for another measurement. I’d ask it to cull the 95% of metrics that mean absolutely nothing to most social media marketers, let alone clients.”

Definitely, every public-facing organization wants to know what’s being said, online and on-social, about its (and competitors’) brands, products, and people. In social-media measurement, too much information is a distraction, just as too little is, per my missing sentiment measures.

There’s plenty to social intelligence beyond sentiment, so here I’ll refer you to Steve Rappaport of the Advertising Research Foundation, whose Digital Metric Field Guide will be coming out soon. Steve provided me a preview of the guide, which recommends 197 metrics, based on consultations with 30+ “authoritative metrics sources,” citing nearly 150 research studies and reports, with 12 essays contributed by recognized industry experts. For now, you can read snippets on Steve’s Blog, and also check out Steve’s workshop, The Insider’s Guide to Social Media Measurement, which is one of several offered as part of my March Sentiment Analysis Symposium.

That’s a lot of metrics, 197, and we haven’t heard the last on this topic. Some metrics that are useful to one organization seeking to accomplish a certain task will be of no use to another, in a different industry or with a different task. What I referred to as “abstract attributes” however — I gave the examples of engagement, advocacy, and connection — will be universally applicable. (Symposium speakers will cover those topics too.) And in all cases, ability to reach beyond the metrics, to assess and explore root causes, the Why behind the measured What, is of great value.

Reach Through, Reach Beyond

I claimed that text-derived scores carry no explanatory power. For explanations, you have to explore the source material.

Many dashboard products will allow you to reach through to the text — tweets, survey verbatims, product reviews, or e-mail messages — that were the source of measured and reported values. Reach-through is more complicated when you’re working with derived indicators such as advocacy and engagement, but possible nonetheless, and it’s less than highly precise if your dashboard was populated with keyword-reliant technology.

Capable NLP technologies will identify topics and concepts related to keywords — call all that good stuff “features” — and associated relationships and attributes including sentiment. Via these associations, reach-through can be particularly useful. You’ll be able to assess salient information that is topically relevant to your business challenge and explains the stats reported by your metrics and indicators.

Root-cause analysis is an exploratory process rather than a measured quantity, otherwise I’d offer root causes as my fifth and final missing metric. They’re essential, as is social-analytics accuracy and the right choice of metrics, in the search for business insight.

Now go and explore.

The post Social Sentiment’s Missing Measures appeared first on Social Media Explorer.

It Goes Without Saying: From Sentiment to Intent

Seth Grimes — Thu, 18 Oct 2012 17:00:04 +0000

When someone remarks “It goes without saying,” they are about to explain, possibly in detail, something the listener is supposed to already know. Similarly, a speaker “who needs no introduction” is often about to get one. When you’re negotiating with someone who says “It’s not about the money,” the sticking point is probably just that, the money, and that classic dump-your-boyfriend line, “It’s not you, it’s me,” means, “You’re not smart/good looking/caring/interesting enough for me.”

Say What You Mean

Why don’t we say what we mean? Actually, I’d say we usually do. It’s just that our meaning is other than the literal reading. This indirection presents a huge challenge for automated natural-language processing (NLP). Meaning is complicated by sarcasm, irony, idiom and other bits of sentiment, subjectivity and ambiguity that make the language-understanding challenge yet more difficult. Ambiguity is easily illustrated. Is “cheap” positive, negative, neutral or … situational? I’m happy to find a cheap airfare because I’m getting the same service as others at a lower price than most, but a cheap appliance may suffer from quality issues. Context matters.

Clues would be great but aren’t easily available. 13 Little-Known Punctuation Marks We Should Be Using, listed by Adrienne Crezo in the mental_floss blog, would help, but frankly widely-and-quickly accepted new conventions (think of emoticons and Twitter hashtags) are rare. Fortunately, people are good at reading between the lines to discern and decode obscured meaning in everyday speech and writing. Machines have to work smart to match human capabilities. Idiom is a contextual indicator, and so are cooccurrence of words and terms and a variety of other algorithmically discernible clues. Software, in the form of computational linguistics, is catching on.

Teaching Computers to Understand

Can we teach computers to reliably and accurately understand human communications? My career for the last ten years has been premised on the belief we can, via text analytics, sentiment analysis, and statistically powered search for patterns. I’m encouraged by innovations in machine learning, particularly supervised methods that build predictive models from training sets. Leading-edge solution providers such as Converseon, a social-media agency, have invested significant resource in building training sets for a variety of business domains (e.g., hospitality, consumer electronics, and financial services). In their case they’ve done this via crowd-sourcing, which helps extend human analyses to Web scale. (Disclosure: I’ve done a small amount of paid consulting for Converseon, which is a sponsor of my next Sentiment Analysis Symposium, October 30 in San Francisco.) Active learning is gaining traction: machine learning extended with human correction of machine-classified results. It’s a recently announced addition, for instance, to SAS Text Analytics capabilities.

I learn a lot from experts, so I recruited social psychologist and social-business pioneer Kate Niederhoffer as one of the keynotes at the October 30 sentiment symposium. Kate’s talk is titled “Sentiment Driven Behaviors; Sentiment Driven Decisions.” The idea is that effective descriptive and predictive methods, drawing from social and enterprise-feedback sources, applied in business settings, will meld language analysis with psychological profiling and data drawn from the variety of available sources. My friend Tom Anderson of Anderson Analytics has been applying a “triangulation” strategy years, matching text-extracted insights to profiles and numerically-coded survey response data.

Yoda Speak

Many researchers and practitioners share a fascination with language (and with data), with text (and speech) that communicates not only facts and opinions but also clues regarding motivations. Consider the example of Yoda-speak, as described by Andrew McAfee in When Did Yoda Start Writing CEO Speeches?:

“Instead of saying ‘Our costs are rising,” [business folk will] say ‘Things are not great right now, from a cost perspective.’ What’s going on here, I suspect, is that they know the overall sentiment they want to convey. In this case, it’s not a good one; costs are rising. So on the fly they construct a sentence that leads with the sentiment (things are not great) and backloads with the reason why (from a cost perspective).”

I don’t completely agree with McAfee’s interpretation of his example. Many readers would see “our costs are rising” as neutral rather than negative. Expressions, and not just words, may be ambiguous. I ran a quick poll I ran back in June that showed this effect: What is the sentiment of “I purchased a Honda yesterday”? Twelve of 22 responses, 55%, rated that statement positive while the rest saw it as neutral. You need context, or information that’s more explicitly conveyed, to fully understand meaning. The Yoda-speak phrasing of McAfee’s example doesn’t provide context but it does reinforce the negativity of, “not great right now.”

From Sentiment to Intent

Context is king, as is interpretation in light of business goals. Here rises the matter of intent or, more formally, intentional analysis. We wish to get at nuance, to distinguish feelings from hopes from plans. From what you say, we want to know what you plan to do. This is a topic I blogged last winter, in an article on sports and political odds-making via the SentiBet system. Start-up Aiaioo Labs scores intent signals that include purchase, wish, inquire, direct, complain, sell, compare, and quit. As-a-service provider OpenAmplify similarly seeks to get at intent signals within broader sentiment expressions.

Maybe you share my fascination with language. Maybe you’re simply interested in business use, in better business decision making. It doesn’t quite go without saying that there is immense potential business value in sentiment and intent signals in social, online, and enterprise data. That said, I hope I have made clear that software solutions (and crowd-sourcing!) are poised to help you discover that business value, to automate understanding of meaning, sentiment, and intent.

The post It Goes Without Saying: From Sentiment to Intent appeared first on Social Media Explorer.

Metavana Mix: Social Complexity, SparkScore Simplicity

Seth Grimes — Fri, 20 Jul 2012 12:00:56 +0000

Metavana is a new-on-the-scene semantic-analysis vendor whose core science invokes a supposed universal descriptive pattern, the Maximum Information Principle. MIP, Metavana explains, describes the distribution of galaxy sizes and, as exploited by Metavana’s software, the distribution of multi-term, natural-language “n-plets.”

Interesting, but there’s plenty of computational-linguistics and semantic-science mojo in a host of established, competing text and sentiment analysis offerings, developed by smart people. The real question is this one: Does MIP make for great “solutions that measure customer satisfaction,” capable of “taming the chaos of the social Web”?

The answer is unclear, not yet established, despite an argument by authority, that the company was “founded by a renowned physicist,” Dr. Minh Duong-van (who speaks about the science and technology in a long video posted by Dana Stanley at his Research Access site, which Metavana CMO Romi Mahajan blogs for), and given only a couple of customer wins.

One very notable industry partnership does speak in Metavana’s favor, an alliance with Satmetrix to produce a simple, consumable number, a SparkScore, that seeks to lend new life to SatMetrix’ venerable but tired Net Promoter Score (NPS). Metavana, however, is only the latest of several text analysis companies that complement NPS with text-extracted information — Attensity, Clarabridge, Kana, Lexalytics, and others have supported the practice for years — and I even have a speaker lined up on the topic for the next Sentiment Analysis Symposium (which I organize), Bill Tuohig from J.D. Power and Associates. Yet the SatMetrix link is significant and will certainly help Metavana make sales. Read Dana Stanley’s GreenBook blog interview with Satmetrix CEO Richard Owen for detailed background. It’s interesting reading, but do keep in mind that Stanley ~~apparently~~ has a business connection with ~~Metavana~~ a Metavana insider.

Metavana’s published materials lack detail that would allow for assessment of company claims, and there’s no testable public interface. Further — a bad sign — Metavana co-founder Spencer Trask, a private-equity firm that is seeking to sell a $6 million Metavana stake, is disseminating information that unfairly slurs competitors and contains a number of factually incorrect statements. But getting back to what can be learned, let’s examine my earlier question, restated —

Is Metavana that innovative?

Metavana’s “search for meaning” technology extracts entities, attributes, topics, and sentiment from text. The company compares its Data Feed offering to Thomson Reuters’ streaming of news and financial information (without providing information on content or coverage) and offers (or will soon offer) an as-a-service sentiment scoring engine, which Erick Watson, Metavana director of product management compared, in a June 25 briefing, to ViralHeat’s. Our briefing centered on Watson’s showing me Metavana’s self-service social-intelligence application. It’s a graphical interface for exploration of social-harvested reviews and opinions, with the ability to associate polar (positive/negative) sentiment to star ratings and hooks for a pending influence measure. See it for yourself in a demo recorded by Watson.

The MIP approach seems quite reminiscent of natural-language understanding efforts that date to the late 1950s. Metavana’s application of the Maximum Information Principle (described in a company white paper) applies the same term-frequency-as-an-indicator-of-significance seen in Hans-Peter Luhn’s 1958 “The Automatic Creation of Literature Abstracts.” Other, latter-day technologies break text into n-grams, which are multi-word sequences. Metavana uses the words “singlet,” “doublet,” and “triplet” where a linguist would refer to an “n-gram” (n=1, 2, or 3), and then Metavana assembles its n-gram-equivalents into unordered sets, borrowing the term “n-plet” from the physics world of the company’s founding scientist. Frankly, I don’t see a meaningful, practical difference between Metavana’s “n-plets” and the sets of terms that variations of a well-established probabilistic technique, latent semantic analysis, will adduce as typical of different text clusters. Documents that contain “dog,” “cat,” and “goldfish” might typify one cluster while “mustang,” “prius,” and “escalade” typify another.

According to President Michael Tupanjanin, Metavana applies machine-learning technology to refine classifications, and also creates models specific to different business domains such as smartphones, printers, hotels, and airlines. (Tupanjanin presented a 5-minute lightning talk at the May 8, 2012 Sentiment Analysis Symposium.)

There must be nuances to Metavana’s mixture-decomposition approach, but is the company’s approach better? I don’t know, but I do know that it’s not as good as asserted. Michael Tupanjanin states, “the algorithms that we have written have taken accuracy to a whole new level, up to over 95%.” This claim is repeated by founder Dr. Minh Duong-van, who puts Metavana sentiment resolution accuracy is “95, 96 per cent accurate,” in the video I linked to above. It is not supportable. Metavana’s measurement method is flawed, as I described in Never Trust Sentiment Accuracy Claims. Do not accept that 95% figure, but refocusing, let me restate the main point of that article, that —

Accuracy isn’t enough

The insights delivered by any worthwhile data analysis have to be useful and consumable. In my opinion, sentiment analysis that assigns a positive/negative/neutral score, at the document, sentence, or phrase level — that’s what Metavana does — are rarely sufficiently useful. Those crude tools are accurate enough to help you hit the “broad side of the barn” — I’ll give them that much — but they’re of little help when your business decision-making requires guidance that is highly specific, when the broad side of the barn isn’t good enough.

Metavana targets the customer-experience space. Better tools in that space (and in others), for instance from Clarabridge, extract sentiment at the feature level — for names of companies, products, and brands and for features such as, using hospitality as an example, a hotel’s cleanliness, staff friendliness, location, comfort, and value — and further analyze sentiment according to categories such as emotion (e.g., angry, sad, happy), or even in Crimson Hexagon‘s case, categories set up by the end-user business analyst, which may be much more useful than positive/negative assignments.

Metavana’s materials, devoid of detail, do not explain or justify ing the software’s apparent limitations. Given Metavana’s ability to extra topics from documents, why is the company content with cruder sentiment analysis, capable only of sentence- and phrase-level resolution and of extraction of only positive/negative/neutral sentiment? Should it boast about high-precision but crude sentiment analysis when rivals resolve sentiment at the more-granular (and higher-recall) feature level, and of polar sentiment scoring when others can handle arbitrary categorizations, such as emotion categories, that are more business-outcome aligned? But maybe all this detail doesn’t matter. Perhaps the answer is that, so often and in Metavana’s case —

Simplicity sells

You can check out the SatMetrix-Metavana SparkScore via a demo site. According to that site, “The SparkScore Sentiment Engine, powered by Metavana, combines the methodology of Net Promoter with customer sentiments from the social graph to deliver an integrated view of customer experience without the noise.” Metavana compares the SparkScore with the Klout score, which Metavana (very justifiable) derides as limited and simplistic, but which provides an attractively simple mechanism for scoring social influence.

In the end, Metavana’s messaging is similarly, attractively simple: Authority, science, and in SparkScore, a special insights-delivery capability and route to market. Will a reliance on idiosyncratic accuracy claims sell product? Blogger Vikki Chowney of eConsultancy is skeptical is skeptical of SparkScore, and so am I. I further see the apparent crudeness of the Metavana’s sentiment resolution as a competitive disadvantage that may discourage adoption of Metavana’s as-a-service and social-intelligence workbench offering, that is, unless the company fights a price war to wrest the quick-and-dirty market from rivals such as ViralHeat and the plethora of low-end social-intelligence dashboard vendors.

Will a haze of science and authority nonetheless create magic that convinces other skeptics? We’ll see.

The post Metavana Mix: Social Complexity, SparkScore Simplicity appeared first on Social Media Explorer.

Never Trust Sentiment Accuracy Claims

Seth Grimes — Tue, 17 Jul 2012 13:00:33 +0000

Sentiment analysis plays a key role in social intelligence (a generalization of social-media analytics) and in customer-experience programs, but the disparity in tool performance is wide. It’s natural that users will look for accuracy figures, and that solution providers — the ones that pretend to better performance — will use accuracy as a differentiator. The competition is suspect, for reasons I outlined in Social Media Sentiment: Competing on Accuracy. Per that article, there’s no standard yardstick for sentiment-analysis accuracy measurement. But that’s a technical point. Worth further exploring:

Providers, using human raters as a yardstick, don’t play by the same rules.

It’s fallacious that humans are the ultimate accuracy arbiters anyway. Can a machine in no way judge better (as opposed to faster or more exhaustively) than a person?

This focus on accuracy distracts users from the real goal, not 95% analysis accuracy but rather support for the most effective possible business decision making.

To explore —

Human benchmarks

We benchmark machine performance, on purely quantitative tasks, against natural measures: Luminosity according to a model of the sensitivity of the human eye (candela), and mechanical-engine output against the power of draft horses (horsepower). But just as a spectrometer measures light of wavelengths unseeable by humans and quantifies visible-wavelength measurements in a way humans never could, and a Saturn V rocket will (or could) take you places an animal could never go unassisted, I believe that sentiment and other human-language analysis technologies, when carefully applied, can deliver super-human accuracy. I believe it is no longer true that “The right goal is for the technology to be as good as people,” as Philip Resnik, a Univ of Maryland linguistics professor and lead scientist at social-media agency Converseon, puts it.

As Professors Claire Cardie and John Wilkerson explain, “The gold standard of text annotation research is usually work performed by human coders… In other words, the assessment is not whether the system accurately classifies events, but the extent to which the system agrees with humans where those classifications are concerned.”

“Agrees with humans”

Note the statement, “the assessment is not whether the system accurately classifies events, but the extent to which the system agrees with humans where those classifications are concerned.”

And consider a company, Metavana, that competes on accuracy, with claims of 95-96% performance on combined topic extraction and sentiment analysis. Metavana President Michael Tupanjanin says the company measures accuracy “the old fashioned way.” According to Tupanjanin, “We literally will take — we recently did about 3,000 quotes that we actually rated, and we sat down with a bunch of high school kids and actually had them go through sentence by sentence by sentence and see, how would you score this sentence?” I praise Metavana’s openness, but this approach is backwards, as we shall see. It assesses whether humans agree with the machine, not whether the machine agrees with humans, per established methods.

According to Erick Watson, the company’s director of product management, the software identifies entities and topics and then further mines sources for sentiment expressions. In the automotive sector, says Watson, the engine identifies expressions “such as ‘fuel efficient’ or ‘poor service quality’ and automatically determines which of these sentiment expressions is associated with [a] brand.” Sounds reasonable, but then Watson wrote me, “Expressions that contain no sentiment-bearing keywords are classified as neutral (e.g. ‘I purchased a Honda yesterday.’)”

I ran a Twitter poll on Watson’s ‘I purchased a Honda yesterday.’ With 22 respondents, 45% rated it neutral and 55% rated it positive. Humans may see sentiment in an expression that contains no sentiment-bearing keywords! Metavana’s summary dismissal of such expressions, coupled to an accuracy-measurement method that restricts evaluation to machine-tagged expressions (the ones the company doesn’t dismiss), inflates the company’s accuracy results.

There’s more to the accuracy appraisal.

Beyond humans

I believe that sentiment and other human-language analysis technologies, when carefully applied, can deliver super-human accuracy. True, we’re years from autonomous agents that can navigate world of sensory (data) inputs and uncertain information in order to flexibly carry out arbitrary tasks, which is what humans do. But arguably, we can design a system that can, or soon will be able to, conduct any given task — whether driving a car or competing at Jeopardy — better than a human ever could.

A first attempt at automating a process typically involves mimicking human methods, but an intelligent system may reason in ways humans don’t. In analyzing language, in particular, machines look for nuance that may emerge only when statistical analyses are applied to very large data sets. That’s the Unreasonable Effectiveness of Data when, per Google’s Peter Norvig, “the hopeless suddenly becomes effective, and computer models sometimes meet or exceed human performance.” That’s not to say that machines won’t fail, badly, in certain circumstances. It is to say that overall, in the (large) aggregate, computers can and will outperform humans both on routine tasks and by making connections — finding patterns and discovering information — that a human never would.

Think of this insight as an extension of the Mythical Man-Month corollary, that “Nine women can’t make a baby in one month.” A machine can’t make a baby at all, but one can accelerate protons to near light speed to create sufficient mass for collisions to result in the generation of unseeable, but inferable, particles, namely Higgs bosons. Machines can already throw together (fuse) text-extracted and otherwise-collected information to establish links and associations that a human (or nine hundred) would never perceive.

Philip Resnik’s attitude, the established attitude, that “the right goal is for the technology to be as good as people,” is only a starting point. We seek to create machines that are better than humans, and we should measure their performance accordingly.

The accuracy distraction

My final (but central) point is this: The accuracy quest-for-the-best is a distraction.

Social intelligence providers often claim accuracy that beats the competitions’. (Lexalytics and OpenAmplify should be pleased that they’re the benchmarks new entrant Group of Men chose to compare itself to.) Providers boast of filtering the firehouse. They claim to enable customers to transform into social enterprises, as if presenting or plugging into a widget-filled social-analytics dashboard, with simplistic +/- sentiment ratings, were the key to better business operations and decision making. Plainly stated —

The market seeks ability to improve business processes, to facilitate business tasks. Accuracy should be good enough to matter, but more important, analytical outputs should be useful and usable, aligned to business goals (positive/negative sentiment ratings often aren’t) and consumable within line-of-business applications.

I’m interested in how your technology and solutions made money for your customers, or helped them operate more efficiently and effectively, or, for that matter, saved lives or improved government services. The number that counts is demonstrated ROI.

Disclosure: Earlier this year, Converseon engaged me for a small amount of paid consulting and was a paying sponsor of my November 2011 Sentiment Analysis Symposium.

And a plug: Check out the up-coming Sentiment Analysis Symposium, slated for October 30, 2012 in San Francisco, preceded by a half-day Practical Sentiment Analysis tutorial, to be taught by Diana Maynard of of the Univ of Sheffield, UK.

The post Never Trust Sentiment Accuracy Claims appeared first on Social Media Explorer.