The Beauty in the Chaos of Conversations: Managing Data and Turning Threads into a Visual Story of Knowledge
This article will cover:
Knowledge and information in an organization can be scattered across many different areas: Email, Slack, Drive, GitHub, and even conversations at the lunch table or by the water cooler.
For example, think about all the emails in your inbox. If you have different email addresses, for personal purposes or for work, it may start getting overwhelming. Or can you tell how many messages are newsletters, order confirmations, or a friend checking up on you? Now, imagine all the data you have on your instant messaging apps, social media, and so on.
If this happens with one person, imagine the amount of data a business manages on a regular basis!
Who Enters, Manages, and Owns Data? The People!
At Commercient, we have a very strong and robust knowledge base. However, as with all organizations that manage complex software solutions, some of the company’s knowledge is within the people. We usually refer to this type of knowledge as, “tribal knowledge.”
Tribal knowledge refers to information that is not documented in files or PDFs, but is intrinsic to individuals within the organization. In this case, it’s the knowledge that people at Commercient possess. For example, one member of the Customer Success team might know specific details about an unusual bug in the sync process and its fix—details that aren’t necessarily secret, but has yet to be documented.
This is the type of information we seek to investigate. We are looking to mine our company’s Zoom Chat group data for technical “tribal” knowledge.
Why? Zoom is one of the main communication channels in Commercient. Besides Google Workspace apps and project management tools, Zoom has proven to be a useful tool for real-time communication for a team spread all around the world.
Every day, more than 100 people share on Zoom what’s going on in their departments and their lives. From customers leaving reviews, new deals being signed, successful project completions, to birthday wishes and photos of the cities they live at and visit, the amount of data, plus the variety, qualified Zoom for this AI research.
Tribal Knowledge: The Value of Conversational Threads
One of the key parts of this type of AI research is understanding the semantic relationships between conversational threads. The position of words and sentences relative to each other helps us to understand the overall picture of the topics being discussed.
For example, we can pinpoint the position of our house, apartment building, car, city, etc. All of these places and objects have a position in our three-dimensional space. In the same way, words and sentences exist in their own higher-dimensional space. The difference is that we cannot see them as we can see, for example, our phone lying on the table.
However, we understand that there are semantic relationships between words. For example, “king” and “queen” are closer in semantic space due to their similar meanings, whereas “king” and “CEO” are less close. However, there is some degree of proximity between “king” and “CEO” because both represent figures of leadership.
Similarly, “car” and “motorcycle” are close to each other in semantic space because both relate to transportation. In contrast, “king” and “car” are far apart because their semantic relationship is not direct; they don’t share a close meaning.
Our AI developers analyzed thousands of messages from our business and technical internal chat groups, where collaborators shared technical knowledge, business insights, news, congratulations, and so on. These messages were mapped to a 256-dimensional space, assigning each thread a vector of 256 values representing its position in the semantic space.
Using various AI algorithms and techniques, our developers assigned labels to these threads. Since we can’t visualize in 256 dimensions, they projected each 256-dimensional vector onto a 2-dimensional plane for easier viewing.
The resulting image below is the focal point of this post. Each point represents an entire conversational thread, projected in a two-dimensional plane, with each color corresponding to a specific topic or label. Threads with similar content are grouped by color—red points represent similar threads, as do blue, green, and so on.
What’s fascinating is that this image encapsulates thousands of interactions, hundreds of ideas, and dozens of feelings. It’s amazing that we can visualize conversations in this abstract yet meaningful way. This 2-dimensional snapshot captures the essence of our interactions, ideas, and emotions, forming a unique, chaotic shape, much like snowflakes or fingerprints.
Data Management and AI: When People And Technology Create Magic
Besides an ERP and a CRM, or any system used to manage data and streamline processes, regardless of the tools and the rise of AI, people are the other part of the equation. The conversations and threads stored in email inboxes and chats hold value that can help businesses get a feeling on not only how much people get done, but also how they work, think, connect with each other and with the company, because all of it is part of the success.
At Commercient, customers say over and over again how the access of data within the CRM or ERP helps them create a 360° picture of sales, marketing, or customer service. Numbers that leadership uses to make strategic decisions. And now, with the rise of AI, Commercient uses it to leverage hundreds of integrations.
