As companies rush to adopt AI, many struggle to define and measure the impact on their operations. Grammarly has driven valuable business outcomes for customers such as Databricks, achieving:
- 70% improvement in written communications
- Marketing team’s editing time cut in half
- Customer Experience team’s time to resolution improved by 25%
However, business leaders want to go beyond improvements in individual departments. They want to understand the overall impact of Grammarly’s AI assistance on their team’s communication, employee productivity, and business outcomes. This raises several fundamental questions: What does it mean to communicate effectively? How can we measure the aspects of communication that an organization cares about? How do we quantify Grammarly’s impact?
Our Data team, in collaboration with Engineering, Product, Design, and Revenue teams, has spent the last few quarters addressing these questions, drawing on our 16+ years of experience in improving communication for over 50,000 organizations, including brands such as Atlassian, Zoom, and Salesforce. This effort has culminated in the creation of the Effective Communication Score (ECS)—the first ever statistical measure of communication effectiveness in the industry.
We’re excited about this score because it gives Grammarly Enterprise customers real-time insights into the overall strength of their organization’s communication, including comparisons with benchmark organizations in their industry, where teams are doing well, and potential areas for improvement. In this blog post, we will explain how we developed and designed the score and share where we’re going next.
How the ECS works today
The ECS is visible to all Grammarly Enterprise admins and account managers. It has three components:
- Overall score: A holistic score that helps organizations understand the overall quality and performance of their communication. This score is the average of five specific metrics (i.e., subscores) that are calculated across all employee communication within the organization. We incorporate industry benchmarks alongside the score, enabling meaningful comparisons across Grammarly’s varied customer base.
A sample ECS report for an organization, including overall ECS score, breakdown of subscores, and Grammarly’s impact on an organization’s communication. When users click on “View detailed score analysis,” they can take a closer look at their score, including the industry benchmark and a detailed score breakdown.
- Score breakdown: This shows the five metrics (i.e., subscores) that contribute to an organization’s ECS, which helps organizations prioritize which aspects of their communication they should work on to maximize the overall score. While the subscores are shown as simplified grades, organizations can look at the underlying numerical subscores that power the grades in the detailed score analysis (shown above).
A view of the score breakdown that powers the overall ECS.
- Grammarly’s impact on your team’s communication: This score highlights Grammarly’s impact on each subscore by comparing the number of writing mistakes before and after applying Grammarly’s suggestions.
A closer look at how the ECS report visualizes Grammarly’s impact on an organization’s communication effectiveness.
A framework for measuring effective communication
While it’s easy to spot effective communication, it’s difficult to define exactly what makes it effective, especially across different organizations. For instance, some teams value thoroughness and detail, while others focus on brevity and precision.
Instead of trying to find the perfect definition, we decided to take a broader approach and brainstorm various ways to define and measure effective communication. This resulted in a list of over 10 different methods. To narrow these options into one cohesive approach, we established three core principles to guide our evaluation:
- Intuitive: The approach should align with how most businesses view effective communication, making it easy for Grammarly customers to understand.
- Impactful: The framework should correlate with positive communication outcomes, such that improving the score would significantly improve an organization’s communication effectiveness.
- Calculable: The methodology must be sufficiently standardized to generate consistent scores across our varied customer base, allowing for meaningful comparison and analysis.
With these principles as our guide, we went through our brainstormed approaches and eliminated the options that wouldn’t work. One potential approach was to consider time savings, based on the idea that communication is more effective if it makes employees more productive. For example, if an employee writes 12 emails (instead of 10) per hour, that results in a 20% saving of time. However, this approach proved problematic, since more emails could indicate poor communication or performative productivity.
Another approach we considered was the number of suggestions accepted, but this metric potentially conflates overall communication effectiveness with Grammarly usage patterns. For example, imagine if both an English professor and a fifth-grader use Grammarly to write an email. The child might receive and accept 10 suggestions, while the professor, whose writing is mistake-free, receives none. Using “suggestions accepted” as our metric would interpret the child’s communication as more effective, despite the professor’s writing containing no errors.
Ultimately, we settled on one measurement approach that received unanimous support: suggestions remaining. These represent mistakes in writing that Grammarly has identified but users haven’t addressed. In this approach, fewer remaining suggestions indicate better communication (and thus a higher ECS). This approach aligns with our three core principles:
- It’s intuitive: Most people expect better writing to contain fewer mistakes, which our definition captures.
- It’s impactful: As organizations reduce the number of mistakes in their writing, their communication effectiveness improves.
- It’s calculable: We have the comprehensive data to measure suggestions remaining across our varied customer base.
We acknowledge that this approach depends on Grammarly’s ability to accurately identify and correct writing issues. While no system can catch all potential issues, our team of analytical linguists, computational linguists, data scientists, and software engineers continuously enhances Grammarly’s AI. This combination of human and AI expertise provides a highly accurate assessment of writing quality that closely mirrors human evaluation of effective communication.
Transforming our framework into an actionable score
With “suggestions remaining” as our measurement approach, we faced three key challenges in developing an overall ECS:
- How to define writing suggestion subscores that capture effective business communication
- How (and whether) to break down the overall score and display subscores to users
- How to aggregate employee communication data into an organization’s overall ECS
Defining subscores
To create a meaningful ECS, we needed comprehensive writing suggestion categories that captured the nuances of business communication. These categories would also power the score breakdown feature in the ECS report, helping organizations understand how their score was calculated and where they could improve.
By design, Grammarly provides three categories of suggestions for all customers:
- Clarity: The conciseness and efficiency of your team’s writing
- Inclusivity: The respectfulness and inclusivity of your team’s communication
- Correctness: The number of common spelling, grammar, and punctuation errors in your team’s writing
In addition, for Grammarly Enterprise users, we offer advanced functionality to ensure that an organization’s communication aligns with its brand standards. We added two categories to evaluate this:
- Style consistency: How well your team follows the organization’s style guide and writing preferences
- Brand consistency: How consistently your team writes in your organization’s voice and tone, as determined by how well your team’s communication aligns with your organization’s brand tone
These five categories became our communication subscores—the foundation for our overall score.
Presenting the subscores
Next, we had to decide how to present the subscores to Grammarly Enterprise admins and account managers. This required aligning our go-to-market, product, engineering, design, and data science teams, each approaching the question from different perspectives.
After evaluating several approaches, we decided to present the subscores as simplified grades: “weak,” “good,” and “great.” This allows Enterprise admins and account managers to quickly grasp the basis for the numerical ECS and identify areas of improvement. We also provide raw values in the report for those who want to dive into the specific numbers.
A zoomed-in view of a sample ECS report that shows how we present the subscores to Grammarly Enterprise users.
Aggregating employee data
With our subscores defined, we had to determine how to aggregate all employee data into an overall score. Specifically, we needed to decide whether to weigh all employees’ communication data equally or give more weight to employees with higher writing volume.
In collaboration with our internal business teams, we consulted customers and discovered that enterprise organizations want a holistic, balanced view of their communication effectiveness. Therefore, we decided to weigh the communication data of all employees across all apps and domains equally when calculating the ECS—preventing high-volume writers from disproportionately influencing the score.
Calculating the ECS
With these decisions made, we built pipelines that compute the ECS using the following steps:
- We examine suggestions remaining across all employee communication in an organization. To effectively measure this, we look at suggestions remaining per 1,000 words for most categories. For inclusivity suggestions, we use suggestions remaining per 100,000 words (since these issues are rarer, but equally impactful).
- We group the identified suggestions into the five subscore categories: correctness, clarity, inclusivity, style consistency, and brand consistency.
- For each category, we calculate the average score (i.e., the subscore).
- To allow for meaningful comparison, we translate the subscore into a percentile by comparing the organization’s subscore against all Grammarly Enterprise users in the same period.
- We average the five percentiles, and the resulting number is the ECS.
Computing Grammarly’s impact
To calculate Grammarly’s impact on improving communication effectiveness, we measure the reduction in mistakes (i.e., suggestions remaining) with and without Grammarly. Specifically, we:
- Aggregate suggestions across four categories: correctness, clarity, inclusivity, and style consistency
- For each category, we track the number of suggestions remaining before and after applying Grammarly’s suggestions. Similar to ECS, we calculate this per 1,000 words for all categories except inclusivity suggestions (which is per 100,000 words).
- Display the results from the previous step in a bar graph with the following metrics:
- Mistakes without Grammarly: The average number of suggestions remaining for that category before using Grammarly
- Mistakes with Grammarly: The average number of suggestions remaining for that category after using Grammarly
Here’s how we show Grammarly’s impact on an organization’s communication effectiveness.
ECS outcomes and next steps
We’ve rolled out ECS to thousands of Grammarly Enterprise customers who already see its value. The score has become a point of emphasis during customer conversations, with many customers citing ECS and its insights as key factors in the decision to expand their relationship with Grammarly. Additionally, it has helped customers better understand their baseline communication performance, their opportunities for improvement, and how they compare with industry benchmarks, which has empowered customers to measure the value of AI and know the impact of Grammarly.
Looking ahead, we’re committed to making ECS even more actionable. We are exploring ways to equip organizations with greater insight into how their teams communicate across specific applications and domains. Our vision is that organizations will be able to leverage this functionality to track and analyze communication performance across their most critical business apps and platforms.
We’re excited about AI’s potential and are trailblazing new approaches to proving Grammarly’s ROI to our customers. If you’re passionate about bridging the gap between advanced AI capabilities and measurable business outcomes, consider joining our team. Check out our job page here.
