Tisane Labs, language, and everything else
official blog of Tisane Labs
We released a Tisane Moderation plug-in for Cisco Webex. The integration plug-in is now live in Webex App Hub.
Install in a couple of clicks and moderate your chats, preventing PII leakage and HR policy violations.
The Webex plugin joins the growing collection of Tisane’s adapters for Slack, PubNub, Maltego, Zapier, Make, Pipedream, and other popular platforms.
Nima by Tremau is a powerful end-to-end platform to manage every aspect of the moderation process, from automatic and manual screening to ensuring compliance with legal requirements across the globe.
Tremau developers have built an adapter seamlessly linking Tisane to Nima. Beyond the basic functionality, Tremau’s Tisane integration can track conversation context for detection of complex, multistep scenarios like grooming and fraud.
Tisane supports detection of multiple types of problematic content, such as cyberbullying, hate speech, attempts to get users off the platform, allegations, sexual advances, suicidal ideation, spam in reviews, and more. Additionally, Tisane provides standard NLP functions like topics, entity extraction, sentiment analysis, and language identification. Tisane supports 30+ languages, with new languages on the way.
Reach out to Tremau and Tisane Labs to learn more.
Tisane has been around for half a decade. It stands apart from other text moderation tools with its architecture. We were shy about our unorthodox approach in our early years; now we have enough traction and confidence to come out.
Today, most text moderation APIs are either machine learning classifiers or rule-based engines looking for keywords. Tisane is neither.
Under the hood, Tisane is a sense disambiguation platform. It receives text as its input, and produces a knowledge graph. The nodes of the knowledge graph contain word senses rather than words; that is, a combination of a lexical item and its interpretation in the current context.
Then, labels are generated based on the combination of graph node constraints. These labels can be anything: entity types, sentiment values, or alerts about problematic content. Think of it as spaCy-like multipurpose engine, but based on a monolithic, interpretable architecture.
tl;dr Because the mainstream solutions are broken in too many ways to count.
The simple keyword or blocklist-based solutions (aka clbuttic) may work for simple tasks in low-traffic communities but will make managing anything bigger or more complex a living hell. (Embrace your inner adolescent and read more with examples.) Most importantly, the bulk of issues do not involve “bad words” but “bad combinations” of neutral words.
Which is why machine learning solutions took over in late 2010s, with the promise of a more nuanced and context-aware solution. While the new paradigm did provide new ways of catching unwanted content, it also created additional issues. Perhaps, the biggest one is adding to the exponentially growing antagonism toward social networks. With the approach often falling back to a bag of words, utterances like “I am a black sex worker” may be perceived as racist. Tweaking and balancing would improve the accuracy somewhat, but not the trust of users whose messages were blacklisted with no explanation.
It is expensive and difficult enough to develop ML classifiers for the major languages. But the trickle-down economics for languages with less data worked even worse. Suffice to say that even juggernauts like Facebook did not deploy a classifier for a language with as many speakers as Hindi as late as in 2021.
But that’s not all.
Just a few days ago the former Google Safety Lead, Arjun Narayan, named the black box aspect of AI one of its biggest issues. It’s hardly new or controversial. When in late 2022 Silicon Valley investor Sriram Krishnan spoke about his transparent moderation proposal, Mark Cuban countered that the roadblock is AI being black box and “hard to audit”:
Opaqueness, as we know, is baked in machine learning. The best explanation ML can give is pinpointing the nodes that influenced its decision. The rest is untraceable, a bajillion-dimensioned matrix multiplied by another bajillion-dimensioned matrix.
If Kafka were writing his Der Process today, he would absolutely feature an equivalent of confidence values. Because being judged by a black box is not scary enough. It has to be a combination of a black box and dice.
The whole idea of pigeonholing human behavior into “naughty” and “nice” may be suitable for Santa Claus, but trust & safety people need to do better than that. And only a sadist Santa Claus would combine his own judgement with a roll of a dice. “Well, it’s 73% confidence value… yeah, probably naughty. If it were 72.98%, then the elves from Mechanical Turk had to take another look.”
We saw customers who hinge their entire moderation process on the confidence values; with both their classifiers and human moderators tending to err on the side of “guilty” (as a character in another Kafka’s story would put it, “guilt is never to be doubted”), the posters had to adapt their habits in a textbook example of leaky abstraction. They would obfuscate words like “fascist”, follow up with puzzled posts asking why they were banned after pointing out racism, and get banned again.
Industry leaders like Grindr’s Alice Goguen Hunsberger say that it’s not uncommon for an ML model to have high confidence but be wrong, and the human moderators must not be swayed by it.
That sounds like (pardon the pun) a lack of confidence in confidence values to me, and Alice is far from being alone in it.
The confidence value is then a security theater, introducing dangers like automation bias. In the perfect world, human moderators are PhDs in ethics debating big philosophical questions; in the real world, they are underpaid students strongly encouraged to perform their tasks as fast as possible. “Computer says no” more often than not means the human moderators will say no, too.
Clearly, finding arbitrarily defined “nasties” in text is more than just a “classification task”. A proper platform used to punish and ban people needs to have the decency to explain itself. It needs to adapt to the ever-changing and ever-expanding needs. It needs to make do with little to no training data. And it needs to be serviceable and updateable without the need to retrain huge models, wasting resources and risking regressions.
In other words, a black box paradigm is a poor fit for text moderation engines. Not to mention that the logic of a giant switch case statement is a bad fit for curve fitting in general.
Yes, trust & safety people found ways to adapt to the deficiencies of the current black box engines. But look around you. The discontent is everywhere, regardless of the political and social views. And even top tier communities like Twitter still fall back to keyword-based decision flow. Laws are being drafted to change the situation and right the perceived wrongs. World authorities in trust & safety talk about the tools being inadequate for their needs. While the objections target the process in general, the shaky technological base is at the heart of the issue.
“You say ML is a bad fit and keywords are inadequate. What did you build then?”
Tisane is a knowledge graph solution, but much more granular than most its peers. Architecturally, it can be viewed as a symbolic counterpart of seq2seq. Our language models contain a rich set of morphological, syntactic, and semantic features, merged from a variety of sources. Because the runtime is symbolic and interpretable, we have full control over the flow, and can introduce new device at any stage.
Of course, building this kind of engine together with proprietary language models for it is easier said that done. If you’re a bit older, you may be familiar with mostly aborted attempts to build this kind of a multipurpose contraptions. As anyone familiar with this architecture will tell you, it’s basically a hoop-jumping marathon with contestants dropping like flies. What only 5% of those who tried building it will tell you is that once you’re done with the Shawshank Redemption-style sewer crawl, things become much easier, and adding new features that take ages with other engines gets trivial.
So what did we get with our off-the-beaten-track approach?
The primary requirements are:
Tisane both pinpoints the problematic fragment and, optionally, provides a human-readable explanation why it thinks the fragment is problematic.
How is it possible? Much like other parts of the language building process, Tisane design workflow utilizes a combination of automated techniques and expert input with a highly efficient formalism. (The runtime is symbolic, but the designer platform is a literal smorgasbord!) In the example above, the “unwelcome person” may be any kind of unwelcome person, be it “liar”, “cheater”, “thief”, “coward”, etc. The query is on the specific nodes in the parse tree; meaning, it can also be interspersed with stray nouns, adverbs, and more (assuming the sentence is making grammatical sense or close to it, of course). The queries are across languages; as one such query covers thousands of combinations in one language, multiply it by the number of supported languages for the total coverage. Queries can be quite specific and convoluted, like tweezers for data extraction.
With Tisane, a human moderator does not need to scan a huge wall of text and scratch their head trying to figure out why “computer says no”. Tisane provides the reasoning behind its decisions, according to the requirements of EU Digital Services Act and other frameworks. The moderator job then is reduced to a sanity check, rather than trying to figure out alien logic while acting as a judge and an executioner. If Tisane messed up, it’s not going to cover its tracks; it’ll make it obvious.
If it isn’t enough, and the language model designer wants to drill down into the parsing process, there is a special web-based debugger which replays the process step by step and generates debugging info for the structures involved.
It is well known that 60% of software life cycle costs come from maintenance. With the black box ML engines, the need to retrain the entire model every time pushes the costs even higher. Retraining a model in an attempt to eliminate a flaw may introduce regressions, with the consistency difficult or even impossible to control. A bit like the Forrest Gump’s box of chocolates, but of the kind nobody wants.
In contrast, Tisane’s processing flow is predictable. That is not to say that regressions are impossible; it is software, after all. But all conflicts can be tracked and ultimately resolved. Generating a language model from the design tables (“compilation”, as we call it) takes minutes on a very standard machine without GPUs (or seconds for the metadata).
When everything is structured, adding more features to the word sense nodes becomes trivial. For example, with hate speech, most engines already moved to tag that it’s hate speech (rather than a generic “toxicity” label from mid-2010s). Tisane takes it one step further and also adds the sub-type of bigotry (e.g. Islamophobia, homophobia, xenophobia, etc.).
How did it come into being? A customer asked whether it’s possible, and it took a couple of hours to map the root nodes; the rest was propagated automatically.
The requirements multiply faster than anyone expects, and so new features are not added just for the heck of it. We don’t have to build a new model every time, just add a couple of features to the model.
Over its analysis, Tisane plays numerous what-ifs and is far more resource intensive than a simple syntactic parser or a part of speech tagger; in fact, these tasks are included in the analysis process. But it doesn’t come close to the CPU and RAM consumption of machine learning moderation engines.
Tisane runtime is built on cross-platform POSIX C++, with native editions for Linux, Windows, and maybe other OSes in the future. It is small enough to be deployed on device, and some of our customers do just that.
Last but not least, Tisane is an energy efficient C++ application, with C++ consistently rated as one of the most energy efficient programming languages. And while it may be disappointing for AWS, Azure, and Nvidia, we know that we contribute way fewer carbon emissions than our energy-hungry peers, and save a lot of hosting expenses for our on-prem customers and ourselves.
Tisane’s language model design module is built to combine input from different sources: fuzzy matching, guided import, and manual expert input. (Plus generative AI, see closer to the end.) While we can use them for benchmarking, we do not require tagged datasets to learn from; we may utilize sources like Wikidata (as a starting point), augmenting them with expert input in more ambiguous cases.
This opens vast opportunities when tackling niche and low-data languages. A training set does not have to come in one package or (again) a black box. We don’t have to wait decades until enough data “trickles down”.
While for most trust & safety tasks the logic can be found in one sentence, or maybe in another one just before it, some cases are more complex.
Consider the following utterance: “Where are your parents now?” On its own, we don’t have enough context to mark it as problematic. It can be anything: two kids talking about their parents or a repairman trying to locate the head of the household. Then consider another utterance: “Can you turn on your camera?” Once again, the utterance alone does not necessarily imply nefarious intent. It may be from a conversation between colleagues at work. However, when the request to turn on the camera follows the inquiry about the location of the parents, it makes a lot of sense to issue an alert.
Tisane makes use of flags that can be set externally or internally to accumulate pieces of evidence when several signals are required.
The same flag mechanism also helps to reduce false positives. For example, in gaming chats, or chats discussing competitions, “Shoot him!” or “We’re gonna kick your a*” are perfectly acceptable and are not indicators of ill intent.
Tisane has a special flag to prevent “competitive language” from triggering alerts, and a mechanism to parse coded language.
When ChatGPT made everyone pay attention to generative AI, we were taken aback like many others, and even worried. But that didn’t last long.
The separation into the design platform and the runtime allows plugging whatever future technologies may emerge into the design platform, which still produces the same format, but with more accurate extraction and more new features.
For example, we’re using generative AI to suggest changes to our knowledge graphs and correct anomalies.
In a way, we’re creating a combination of an “LLM snapshot” and manual adjustments, while enjoying the best of both worlds: deterministic, efficient Tisane and coverage of edge cases that the original Tisane language models may have been lacking. It is also possible that Tisane can be used as a “Trojan horse” to get some generative AI functionality to the enterprise.
We’re no longer asked whether this kind of architecture is viable, or whether it’s more fragile. We have proven it’s viable, robust, and, most importantly, solves nagging issues that other paradigms don’t.
Can a traversable deterministic graph of entities offer unique advantages beyond trust & safety? Tisane is already actively used in law enforcement and national security, where determinism, explainability, and data economics are essential. We are open to other future applications, too.
Have any questions or comments about the topic discussed in this post? Feel free to comment or contact us and we’ll be happy to help!
After a thorough review, Tisane Bot for Slack has been approved. It is now listed in Slack App Directory.
Tisane Bot for Slack is an easy to use moderation aid for Slack communities great and small. Monitor your Slack channels for cyberbullying, insults, data leaks, hate speech, and other HR policy violations. The bot is easy to install and does not require IT expertise.
Tisane Bot comes with all the capabilities Tisane API has to offer:
Read more about Tisane Bot for Slack
Read more about detection of abuse with Tisane
NLP has its own unique challenges in law enforcement & intelligence scenarios. These challenges are rarely addressed by the mainstream NLP frameworks. From the conversations with our law enforcement users, we learned that the use of code words subverting detection efforts is one such challenge.
Oftentimes, offenders or intelligence assets replace important words by seemingly unrelated mutually agreed upon terms. You never know who’s listening, right? Or where the device ends up. Code words are not the same as jargon or slang; it’s a disposable secret language designed to obscure the real meaning.
Once the investigators start guessing what these unrelated terms mean, they can try using keywords. But what if it’s more than a couple of words, or the inventors are particularly creative or supplement it with purportedly introduced misspellings?
What if the misspellings are the auto-correct gone rogue?
Tisane has a solution for that.
In Tisane, text processing is built around concepts (or word-senses) rather than words. The word-senses are clustered in structures called families, each provided with a unique numeric ID. A family contains a set of synonyms and their inflected forms, complete with a set of features. The family ID is crosslingual, as clusters of synonyms are aligned across all languages supported by Tisane, now or in the future.
By default, Tisane’s decisions are made based on the data coming from its language models. However, since the Darjeeling release, Tisane has a so-called “long-term memory” module. The long-term memory module, among other things, allows making changes to attributes at the level of a call, “redefining” the contents of memory, and preserving the accumulated knowledge, if the calling application chooses to do so. As the word-sense is just one of the attributes, it can be redefined on the fly, too. Doing so assigns a new sense to whole categories. As the family IDs are crosslingual, there is no “family ID for English” or “family ID for Japanese”. It’s exactly the same family ID everywhere.
To assign a new meaning to the code words, the settings parameter in Tisane’s processing methods (POST /parse and POST /transform) must contain the section where the redefinition takes place. The concepts matching the redefinition criteria are then assigned the new attributes. To redefine the word sense, we simply need to supply a new family ID.
In your application, you may allow your users to look up and redefine the word sense on the spot, in order to run experiments with information extraction.
Let’s say, a group agrees among themselves that when they will mention a mushroom (of any kind), they will actually mean a spy. It may be a chanterelle, a truffle, or any other type of mushrooms.
We’ll use two methods:
In Tisane, we need to:
2. Look up the family ID of spy. Again, this is accomplished by invoking GET /lm/senses as shown on the screenshot below:
There is a number of senses; looking at the definition attribute, we see that there is one sense that means a “secret agent”. The family ID is 67433. This will be our target family.
3. Finally, we can build our reassignment clause. We want any kind of fungus to match.
Meaning, we need to match all word-senses whose hypernym (an umbrella concept or a super-family, if you will) is fungus (79199), and redefine them as a family 67433 (spy). The reassignment clause is:
{"if":{"hypernym":79199},"then":{"family":67433}}
The reassignment clauses are kept in the assign array under memory.
Let’s test:
Suppose a law enforcement agency is looking to locate any record of items that could have been used in a burglary. We can, of course, manually search for hammers, screwdrivers, maybe other tools. We may find something or miss something.
We can, on the other hand, link all the tools (e.g. hammers, drills, etc.) and fasteners (e.g. nails) to the category of “illegal item”. That does not mean that the item is illegal, of course, but it will cause Tisane to generate a criminal_activity alert.
We need to:
2. Look up the family ID of illegal item. The family is 123078.
3. Build the reassignment clause. We want items that have family ID 34876 as a hypernym, and those with 28191 as a hypernym, to be linked to hypernym 123078. That will still leave their existing hierarchy intact (they will still be “tools” and “fasteners”), but will have a new link. The assignment array is:
[{"if":{"hypernym":34876},"then":{"hypernym":123078}},
{"if":{"hypernym":28191},"then":{"hypernym":123078}}]
Let’s test:
Many of Tisane’s patterns marking personal attacks are looking for discussion participants. The trouble is, with a name alone, it’s impossible to know whether they are participants in the current discussion. On the other hand, the names and the aliases could be available from other sources.
Wouldn’t it be handy if there was a way to tell Tisane that Kewldude1995 and John_Smith are names of the participants, and the attacks on them should be treated as personal attacks?
While it may not be a “code word” problem, the solution is exactly the same: redefining family IDs. This time, based on a string pattern (as these names are not mapped in the language models).
We need to:
2. Unlike in the previous examples, we need to rely on plain string recognition. This is done by defining a regex condition:
{"if":{"regex":"Kewldude1995|John_Smith"},"then":{"hypernym":123887}}
Let’s test:
What if we want to treat all names as discussion participants? Then the condition in the second step is to be replaced by a condition looking for a name:
{"if":{"hypernym":44155},"then":{"hypernym":123887}}
In this post, we’ve shown how to use Tisane to extract information while redefining the language models on the fly. These techniques can be used to tackle challenges like secret language and auto-correct.
If you have any questions, or remarks, please feel free to contact us or connect to us on LinkedIn.
In the next posts, we’ll show how to use the same long-term memory module to detect patterns based on multiple signals, tackling challenges common in applications like detection of child grooming and fraud.
Darjeeling is the name of our 2021 Tisane update. While we worked on multiple parts of the system, the focus in this update is on moving beyond the “sliding window” of one sentence. The current edition adds powerful tools to make the system aware of non-textual content, as well as context coming from different messages.
See below short notes on the new features and changes.
Automatic Language Detection
For those working with multiple languages, it is now possible to use the automatic language detection. In addition to the new method, it can be activated in other methods as well; simply specify * as the language code value. If the likely languages are known, specify them the vertical bar-delimited list of the likely language codes, e.g. en|es|ja.
More Subtypes for Abuse
We added more tags to describe the various shades of abuse. In the instances of hate speech (bigotry type), we now detect the protected class being targeted, generating tags like racism, xenophobia, sexism, religion, and more.
Runtime Redefinition: Codewords & More
Redefining concepts & features on the fly became possible thanks to the brand new Long-Term Memory module. From conversations with the law enforcement users, we learned that codewords are a persistent challenge that subverts detection efforts. Tisane now allows redefining what concepts mean; not just adding word for word, but redefining entire families or even categories.
E.g. if someone consistently means spies when mentioning any types of mushrooms (be it champignons, chanterelles, etc.), there is a new structure in the settings that allows redefining it. Upon learning of the hidden meanings, the users then can request the service to interpret some concepts differently, reprocess the text, and detect patterns that were skipped. Contact us to learn more.
The redefinition module allows redefining both meanings and features. For example, we can define that the gender of the speaker or the interlocutor for languages where these pronouns do not have gender, generating translations with the correct gender. We can assign new categories, reflecting subjective opinions (e.g. link a particular political faction to the concept of “unwelcome person” to gauge sentiment). The runtime redefinitions will then work in concert with the pre-built language models.
Nontextual Context Flags
Another use of the Long-Term Memory module is to define non-textual context. For example, if a meme shows a pile of refuse or an animal associated with negative qualities (ape, pig, snake) and the text overlay names a protected class, the text in itself is not offensive. However, the message is undoubtedly is. It is now possible to set a flag for particular types of situations. These flags will provide context, allowing Tisane to detect more.
Multi-step Detection
For the subtle and gradual kind of abuse, the picture is never clear from just one sentence. The offender may require multiple steps to gain confidence of the victim, or spin a long story, each part of which is neutral in isolation.
We now allow issuing alerts only when several flags fire. While Tisane remains stateless, it is possible to port the context and the accumulated knowledge from the previous messages in the conversation to the subsequent messages.
In-game chats are a staple of many gaming experiences. Players love to chat with each other. But it’s no secret that many game chats suffer from colorful language that often includes profanity and personal attacks.
Even though players may tolerate this to a degree, game studios must take action to clean this up. This becomes even more important as the gaming community diversifies along age, gender, and nationality lines.
The good news is most game studios do try to keep their chats clean. However, the players who write the verbal attacks are quick to adapt. They can easily defeat solutions based on keyword filtering, and even can outmaneuver smarter solutions that require extensive model training or labeling.
So the question is, how can you offer real-time chat that is welcoming to all?
In this post, we’d like to show you how to build a solution that does just that. The solution not only provides a real-time, scalable chat solution (PubNub), but also keeps the chats clean of abuse (Tisane).
Quick Intro to PubNub
PubNub empowers developers to build low-latency, realtime applications that perform reliably and securely, at a global scale. Among other offerings, PubNub provides an extensible and powerful in-app chat SDK. The chat SDK is used in a variety of scenarios, from virtual events like the ones hosted by vFairs to in-game chats. It can also be used to create chat rooms in Unity 3D.
Quick Intro to Tisane
Tisane is a powerful NLU API with a focus on detection of abusive content. Tisane detects different types of problematic contents, including but not limited to hate speech, cyberbullying, attempts to lure users out of the platform, sexual advances. Tisane classifies the actual issue, and pinpoints the offending text fragment; optionally, explanation can be supplied for a sanity check or audit purposes. A user or a moderator does not have to figure what part of the message and why caused the alert. Tisane is a RESTful API, using JSON for its requests and responses.
Tisane is integrated with PubNub. Just like with other PubNub blocks, Tisane can be easily integrated in PubNub-powered applications. That includes the applications using the in-app chat SDK.
Now, the fun part. Let’s start coding! Follow these steps to get your Tisane powered PubNub account setup.Â
Add a function used by the Before Publish or Fire event.
Statistically, the vast majority of violations are personal attacks (attacks targeting another participant in the chat). When the chat is structured with responses and threads, it’s easy to figure out who is the target of the attack. It’s usually the poster of the comment being replied to. If the chat is not structured, it is a poster of one of the recent messages.
Human moderators usually don’t like moderating game chats. It’s less about community building and more about banning users being nasty to each other. And you know what? Maybe they don’t have to. Since Tisane tags personal attacks, the privilege to moderate a post with a suspected personal attack in it, can be delegated to the suspected target(s). If it is not a personal attack, nothing will happen. If it is indeed an insult, then the offended party will have the power to discard the message, or maybe ban the poster. Note that it will only happen if Tisane suspects a personal attack.
The community both saves on the moderator manpower, and the potential offenders behave differently knowing that the person they attack will have the power to punish them. How often does one attack a moderator?Â
Of course, Tisane is not a panacea. It is merely a tool to maintain good experience. Like any natural language processing software, it may make mistakes. It is also intended to be a part of the process, not to substitute the process.
We recommend gaming communities to combine Tisane with low-tech, tried-and-true methods, such as considering whether the users whose post was tagged, is a long-standing member of the community with good record. When possible, drastic actions should be subject to a human review as well.
We also recommend taking different action depending on the context. Death threats or suicidal behavior should be treated differently than profanity. Scammers and potential sex predators should not be asked to rephrase their posts, they need to be quietly reported to the moderator.
As the cautionary tale of Parler shows, lack of proper moderation may bring down an entire community. All it takes is one incident that may raise the prospect of lawsuits or consumer campaigns to delete your app for everyone involved even indirectly. Nor is it the first time: back in 2018, Gab was forced offline over the Pittsburgh synagogue shooting.
It’s not just the United States. Virtually every jurisdiction today has laws to make sure that online chatter does not cause offline issues. You think you can host your controversial website in Russia? Think again. Roskomnadzor (a Russian federal service in charge of censorship of communications) goes beyond banning neo-Nazi outlets. For example, some websites are forced to replace specific Russian swearwords by “describing what they mean” instead, as in a charade. (Literally descriptions like, “a word made of four letters starting with X that rhymes with Y”.)
And if it weren’t for the threat of deplatforming, users leaving, and pressure from campaigns like Stop Hate for Profit, brands are no longer happy to let their ads be displayed near problematic content. When the №1 advertiser on Facebook considers pulling their spending, no deplatforming is needed to get the message across. With the bulk of social media subsisting on ads, whatcha gonna do when no one wants to advertise with you?
A class action suit is basically a courtroom equivalent of an angry mob with torches and pitchforks looking for targets. It does not matter if a party was just marginally involved; if it can be proven that it was involved at all, and can pay, it will likely be named an accomplice.
Suppose there was an incident or a chain of incidents in which people got killed and massive damage was inflicted. The action was planned on a platform built by a small unprofitable startup, hosted on a cloud maintained by a trillion dollar company, and distributed via app stores by other trillion dollar companies. Do you think the lawyers, whose compensation may be contingent on the settlement amount, will miss an opportunity to pick on the big guys? Neither do I.
The reaction of Apple, Google, and Amazon therefore should not be surprising. What’s surprising is that Parler wasn’t deplatformed earlier, given the number of warnings AWS sent (note the obfuscated F-word in the first post; apparently, that’s what the author felt was the worst part of the death threat). Just before the deplatforming, the data dump of Parler’s user content has been obtained, and is now doing circles in the digital forensic community.
Could Parler have survived? They had moderation in place; very loose, but it was there: no porn, no spam, no obscene nicknames. Some observers complained about the policy being inconsistent and arbitrary. For example, a user got banned for mocking Parler, even though it was never against the published rules.
Clearly, what they had in place wasn’t enough. When you are repeatedly told by your hosting company that you have potential law enforcement issues, you better do something about it. Were they paying attention? Variety and others posted conflicting accounts of the former CEO and the shareholders, mentioning discussions about the need to crack down on the darker parts of the gray zone. Whoever said what, it does sound like the subject was raised.
If that’s the case, they knew it was problematic; they were discussing ways to mitigate the issue. But it took too long.
The moral of the story is, don’t let an inconsistent and lacking moderation policy become a trainwreck in slow motion.
Just around the time when Parler went offline, Google banned Element, a messaging app using a decentralized protocol, from its app store. This time, however, it was only Google, and the ban did not last long. According to the firsthand account of an Element team member in Hacker News:
And that was the end of the story.
The sophistication of Matrix.org moderation toolset is impressive, and should be regarded as a gold standard. The creators of Matrix seem to realize that decentralized != chaos. They have a concept of policy servers, and a component that provides portable moderation and, as they say, “impersonal” rules. Contrast it with the approach of “poop emoji in the comments not OK, death threats OK, but sometimes not, and sometimes we’ll just kick out people who troll us”.
Because proper moderation is, like proper law enforcement, consistent and impersonal.
Here in Tisane Labs, we build automated tools to tag problematic textual content. Tisane API is not a panacea by any means, but we work hard to avoid the common pitfalls. Our starting point is the need, not what can be done quickly. Some of the capabilities of Tisane appeared an overkill at first, only to be requested later by the users.
Outlined below are the core design principles of Tisane.
Transparency and Granularity by Design
Every alert the system provides contains a built-in rationale, and the actual fragment that caused the alert.
A moderator, with their human biases, likely overworked, underpaid, and possibly coming from a different culture, should not be required to be an arbiter and a professor on ethics. Nor should they be required to figure out what’s wrong with an non-abusive utterance like “I am a black sex worker”.
With Tisane, their task is reduced to verifying that the system interpreted the utterance correctly.
Consistent Decisions
Wouldn’t it be more democratic if every policeman could apply the laws the way they see fit? After all, they know the reality better than some lawmakers far away, right?..
Shocked? An insane idea, isn’t it?
And yet that’s roughly how many automatic moderation systems function today. Some rely on models trained by underpaid human annotators; others make it even worse and let the model be trained by decisions of moderators.
Do you want to get deplatformed because five of ten of your moderators were fixated on the poop emoji and taught the system to dismiss the death threats to public figures?
Tisane’s alerts are strictly aligned with the provided guidelines. Our built-in guidelines rely on principles today regarded as universal or near-universal. We do not tag content critical of a political faction or a public figure (unless they are part of the conversation) as a violation. We avoid political decisions; the management team of Tisane Labs, as well the rest of staff is made of people with very different opinions, mindsets, and backgrounds.
That said, if your community wants to steer clear of a particular topic (like the Peloton communities avoiding political discussions), there is a topic detection and entity extraction mechanism. Rather than providing a simple classifier, we follow the “Swiss army knife” approach.
Simplicity
We avoid floating point “confidence thresholds”.
It may give the developers a false sense of control, but it doesn’t make the decisions more accurate, let alone transparent. We view it as an equivalent of security theater in the airports.
Privacy and Data Security: On-prem
Sometimes, the conversations in the community may be more discreet. Whatever the policies of the vendor are, whatever jurisdictions they must be compliant with, some communities want to be shielded from changes in the legislation or intrusion into privacy. On the other hand, some communities have specific requirements, and need to construct their own guidelines.
Which is why Tisane has an option for an on-prem single tenant installation, with the ability to adhere to custom policies. Contact us to request more information.
Pricing
Moderation is important, but investing in fences and locks more than the house is worth makes no sense. We realize that moderation is akin to insurance policy, and keep our prices affordable.
To try Tisane, sign up here. The free plan does not expire.
Moderation is no longer a luxury. Even in cases when it’s not a legal requirement, running an online community without a moderation solution in place is akin to driving without insurance or never locking your house.
You may get away with it for a while, but as your community grows, the probability that bad actors abuse your platform is too high to ignore.
Published on Marketwatch and Yahoo! Finance
SINGAPORE, Oct. 13, 2020 /PRNewswire/ — Tisane Labs, a supplier of text analytics AI solutions, today announced a new feature in Tisane API, already available on Microsoft Azure Marketplace and AppSource. With the new feature, Tisane API now allows tagging and extraction of Wikidata entities, complementing the capabilities provided by Azure Cognitive Services and supporting nearly 30 languages.
“Wikidata allows utilizing Wikipedia knowledge in ways never explored before, but there’re not many ways to get Wikidata references from unstructured text,” said Vadim Berman”Now, with the new feature of Tisane API built on Azure, our users can easily obtain Wikidata IDs from Tisane’s JSON response. Imagine being able to annotate text with images, GPS coordinates, important dates, 3rd party references, and whatever the ever-growing and open Wikidata database contains.”
“Microsoft Azure Marketplace and AppSource lets customers worldwide discover, try, and deploy software solutions that are certified and optimized to run on Azure,” said Sajan Parihar, Senior Director, Microsoft Azure Platform at Microsoft Corp. “Azure Marketplace and AppSource helps solutions like Tisane API reach more customers and markets.”
Tisane API runs in the cloud utilizing Azure API Management, with a simple REST interface that can be linked from any popular programming platform today. Tisane Labs provides a range of tailored plans for its clients with the option of a custom installation on-premises and a free plan.
To try Tisane API, visit https://tisane.ai.
For more information, email press@tisane.ai.
The next update after Boldo has to start with C, and so it’s Chai this time (no, corona does not qualify).
It has been a busy year. With more users and new use cases come more feature requests, and we worked hard to implement them.
In the moderation space, it helps to provide a cue why the system classified a message as problematic. Human moderators are often stressed, overworked, and overwhelmed, while natural language processing is bound not to be 100% error-free. Reducing their task to a simple “sanity check” whether the system understood the utterance makes them more productive and more consistent. When the explain setting is set to true, Tisane provides a short snippet describing the reasoning (settings: read more here).
Many communities require users to enter real names. Some users prefer not to, for different reasons. In some cases, there is a need to break down a full name into constituents, such as given name, surname, middle name, etc.
Tisane now provides methods to:
Read more about name-related methods.
Wikidata IDs are now supported at the level of entities, topics, and even some non-entity words.
Imagine deducing geographic coordinates, Britannica article ID, or semantic properties from text that went through Tisane!
Different content formats may have different logic tied to them, especially when the context is lacking. With new requirements, we added two new formats:
search to screen search queriesalias to screen user names in online communities
With more law enforcement vendors adopting Tisane, we were asked to add entity types of interest to the law enforcement, specifically:
weight bank_account credit_card, including subtypes representing the major credit card typescredential, with optional subtypes md5 and sha-1crypto for major cryptocurrency addressesand more. See all the types here.
We also added two more types to tag problematic content:
adult_only, for activities allowed in adult communities but restricted for minors (e.g. consuming alcohol)mental_issues, for signs of depression and suicidal thoughtsAs always, we keep honing and enhancing our language models, as well as throughput and stability of Tisane.