One of the key challenges facing DAOs involves running an effective operation using decentralized structures and procedures. The DAO model of organizing makes them attractive opportunities for skilled contributors, but when organizations fail to operate effectively, they become sick. Symptoms manifest in organizational attitudes, behavior, and culture that influence the organization’s ability to realize its potential. If not managed, these symptoms create negative feedback loops between worsening efficiency and worsening contributor experiences. This will eventually kill the DAO.

To diagnose and mitigate poor organizational health, psychologists developed empirically-driven psychometric surveys to measure the psychological factors that contribute to a healthy organization.

In the traditional scientific literature on predictors of healthy organizations, these same factors have been shown to predict outcomes such as turnover, staffing needs, productivity, performance, client satisfaction, and profit. [1] Predictors include positive organizational attitudes, engaged workers, supportive cultures, aligned values, and clear communication, among others. After reviewing the literature, we concluded that DAO Health can be sufficiently measured using similar psychometrics.

In partnership with the Ocean protocol, talentDAO is introducing our comprehensive DAO Health Survey – an empirically-driven psychometric tool to gain insights into the health of a DAO.

In this post, we provide details about how this tool was developed, how it should be used and understood, and what is next for talentDAO on our DAO Health initiative.

This write-up is organized as follows:

• Operationalizing DAO Health: how we defined DAOs and DAO Health in order to measure them effectively.
• Literature Review: a quick summary of how we conducted our review of the scientific literature to decide what to measure and how to design the survey.
• Survey Design: a brief introduction to the survey, what it is measuring, how to use it, and a few other good things to know.
• The Future of the DAO Health initiative: what is next for this initiative at talentDAO.

The DAO Health Survey is open-source and can be found on Github. Over time, talentDAO will conduct several studies to improve this measurement tool and new versions will be released.

Operationalizing DAO Health

When we talk about DAO Health, what we are really talking about is organizational health.

As researchers at McKinsey put it, organizational health is “...more than just culture or employee engagement. It’s the organization’s ability to align around a common vision, execute against that vision effectively, and renew itself through innovation and creative thinking. Put another way, health is how the ship is run, no matter who is at the helm and what waves rock the vessel.” [2]

It’s an elegant definition, and we would get away using it in any other context, but in order to come up with a way to properly measure DAO Health, we need a properly operationalized definition of a DAO.

After reviewing a wide range of literature on topics related to organizational health, organizational effectiveness, shared leadership, virtual teams, and worker well-being, we landed at the following operational definition: a DAO is a network of contributors coordinating in dynamic virtual teams toward a shared purpose by decentralizing authority and ownership.

With this as our framework, we arrived at the following operational definition of DAO Health:

The DAO's ability to coordinate teams toward a shared vision and objectives as a function of the contributor socio-psychological factors that drive collective productivity, performance, leadership, and individual experience and well-being.

Arriving at these definitions required consideration of the full range of organizational, behavioral, environmental, and structural factors present in DAOs.

Because DAOs come in many different shapes and sizes, some sharing many similar features with traditional organizations and others sharing few, they can be difficult to characterize. The technical conception of a DAO, based on the definition proposed by Ethereum co-founder, Vitalik Buterin [3], can be summarized as “automation at the center, humans at the edges”. Here, the only human tasks are the tasks that automation cannot do.

By Vitalik’s definition, many DAOs today would actually be DOs – ‘decentralized organizations’. But for today’s builders, DOs may not entirely reflect their ideals. The common denominator is the leveraging of blockchain technology. That is, the use of a custom-built protocol to function as the infrastructure for new technological systems. It may simply be that the range of human input needed to run all possible protocols was underestimated in Vitalik’s view. Alternatively, it was not at all anticipated that such strong connections would form within these networks, realizing in communities of active contributors eager to do more than just get rich off the next meme coin. Therefore, we retain the term ‘DAO’ in our definition as we feel it best reflects the current public interpretation regardless if it is a DAO or DO, by Vitalik’s technical definition.

Although they all vary in degrees of decentralization and autonomy, as human organizations, many DAOs operate within similar constraints. For example, the use of Discord as the dominant communication medium for DAOs, despite its criticisms, is largely recognized as standard practice. Similarly, the DAO ideology has created a culture of shared leadership, distributed functions, and high autonomy working environments sometimes dubbed 'permissionless' work.

With this in mind, we defined DAOs under the assumptions of human organizations.

Organizational Design theorist Richard Daft defines organizations as "social entities that are goal-directed, are designed as deliberately structured and coordinated activity systems and are linked to the external environment." [4]

Our view is that this definition of organizations holds for DAOs, albeit with different management models, cultural norms, and technical capabilities.

The activity and goal-directed behavior Daft is describing is the work – or ‘contributions’ in a DAO. These are the activities and behaviors performed to advance the organization's goals.

Our survey focuses on contributors, those doing the work and actively participating in the DAO’s goal-oriented activities. This includes paid and unpaid contributions as well as contributions from those with different membership statuses. However, this definition excludes people with DAO membership by way of financial backing or holding governance tokens solely, as well as presence alone in an open communication platform like Discord.

Unlike voting behavior or event attendance, the state of the contributor experience is not something that can be captured on-chain. If we aim to measure the health of a DAO, we must supplement on-chain activity metrics with off-chain “contributor metrics”.

We refer to contributor experience as an overarching term that encompasses a breadth of varying factors, each with its own moderating effects and various outcomes at each level in the organization.

Contributors provide talent, the collective human capabilities that add value to the organization and make up the working groups of the DAO, also referred to as 'guilds' or more generally, 'teams.'

From this perspective, there are three organizational levels our tool focuses on: individual, team, and organizational. We discuss the relevance of this classification further in the section on Survey Design.

DAO Characteristics

While each level is important in its own right, the team level is an especially important distinction for DAOs. Not only is there evidence to support that the effectiveness of an organization is largely influenced by the effectiveness of its teams [5], but in a DAO, the default modes of teamwork are decentralized and highly virtual.

This doesn’t mean DAOs don’t ever coordinate in proximity. But the technology underlying DAO governance and financials is unrestrictive of time or space. This opens up all possibilities of distributed coordination from local to global geographies. Yet regardless of geography, when two or more people in a DAO work towards common goals, they use computers and the internet as their primary tools to achieve them.

This results in DAOs with a tendency to be:

• high in temporal distribution — teams are geographically and temporally distributed.
• high in boundary-spanning — teams work cross-functionally and cross-culturally.
• dynamic in lifecycle — teams may be teams for short periods or long periods.
• dynamic in member roles — roles in teams shift depending on the current needs. Leadership is also dynamic. [6]

However, like any other organization, all DAOs are unique in their own ways and fall somewhere on a spectrum between pure DAO — high virtualness, dynamism, and decentralization [referring to both leadership and governance] — and traditional organization.

Traditional organizations may have recently adopted remote work, but they also tend to be geographically and organizationally centralized and with members in contractual employment agreements with the firm.

Decentralization of authority is another key characteristic of DAOs that makes them unique. Again, the exact model a DAO subscribes to will vary, but part of the ethos of being a decentralized autonomous organization is that no single party is in control. Without this feature, DAOs are simply high technology organizations on the high end of the virtualness spectrum.

In a similar vein, DAOs are also open source organizations. By this, we do not mean that all intellectual property (IP) developed by DAOs is made freely available — although it often is — but that the organization itself is open to anyone to join, contribute to, or fork.

DAOs openness has spawned a new kind of working modality akin to independent contracting and emphasized by the frequent utility of bounties. This kind of one-off project-based work arrangement as an entry point to the organization is part of what makes DAOs highly dynamic. Contributors can pop in and out of the organization freely, and will often contribute to multiple DAOs at once.

It is with this understanding that we arrived at the operational definition of the DAO as we initially described: a network of contributors coordinating in dynamic virtual teams toward a shared purpose by decentralizing authority and ownership.

One of the core benefits of this view is that although DAOs have distinct characteristics, they still share many similarities with traditional organizations. Of the most significant is the use of teams and the fact that human capital acts as the mediator to the DAO achieving its goals. Therefore, we should still expect psychometrics adapted from traditional organizational settings to be applicable in the context of a DAO. The psychological factors still exist, albeit with likely different outcomes and baseline levels.

Thus, we arrived at our operational definition of DAO Health: the ability of a DAO to coordinate teams toward a shared vision and objectives as a function of the contributor socio-psychological factors that drive collective productivity, performance, leadership, and individual experience and well-being.

With a now operationalized definition, we began our research into the psychological factors associated with these outcomes and how those have been previously measured.

Literature Review

The first step of our research initiative was to understand the current body of research in order to first operationalize DAOs and DAO Health. The second step would be to determine which psychometrics were appropriate. We reviewed 25 research papers, 11 of which were meta-analyses on subjects pertaining to organizational health, organizational outcomes, teams, virtualness, information sharing, leadership, and performance. To determine the psychological factors appropriate to measure, we evaluated an additional 24 psychometric measure validation studies on various factors for contextual fit. In some cases, multiple versions of scales were reviewed and added to a repository of scales for review.

Expert Panel

Four organizational scientists at talentDAO formed an expert panel of reviewers which served to a) deliberate on operational definitions, question selection and revision, and b) follow an item elimination procedure to reduce survey length to a practical standard of ~50 questions.

The expert panel began with a survey of 126 questions measuring 32 different psychological factors and ended 3-rounds of voting [7] with 50 questions measuring 23 factors. We approximate the full survey to take ~20-25 minutes to complete.

Survey Design

The main goal of this survey is to gain an understanding of the health of the organization so that DAO leaders can take corrective action to positively influence the contributor experience and resulting outcomes for the DAO. To do this, we mapped each factor according to its theoretical underpinnings and attempted to visualize the ways in which they ultimately contribute to organizational outcomes.

Understanding the measures

From a psychometric perspective, each structural level [individual, team, or organizational] acts as its own class with certain applicable properties. Sometimes these properties have applications at multiple levels. However, measures used and insights gained at each level for the same psychometric property may be inherently different. For example, communication at the team level may refer to the interactions between team members, but at the organizational level, it could refer to the messaging and information received from the core team. [8]

We measure several factors at each level, knowing that some of them overlap in their domain coverage.

The table below lists each factor with a brief definition, clustered in its level of analysis. We will expand on each one of these in-depth in future blog posts. For now, feel free to hop over to the talentDAO Discord server at any time to get answers and support on interpreting results.

Deploying the survey

DAOs are in their early stages and are constantly growing and evolving. If at all possible, it is best to conduct this survey at least twice a year. However once a year is sufficient to track changes over time.

It is generally good practice to add additional questions in the survey for open feedback [qualitative responses]. These can be general feedback questions or specific matters of concern at the DAO. For example, “Do you have any additional feedback about this DAO?”. Open feedback questions on organizational surveys like this tend to be some of the most insightful data sources.

Including some demographic items in your survey is a good idea as well. Data points that you can pivot and slice on can have a big impact on how that data is interpreted, such as examining how individual opinions and perceptions differ across groups within the DAO.

While we often think of race/ethnicity, gender, age, etc., as demographic criteria, we encourage expanding this view to include any criteria that may influence results in the survey including part-time or full-time contributor status, financial investment status, core team status, bounty hunters vs active contributors, education experiences, previous work experiences, web3 tenure, etc.

Assessing results

Our current recommendation is that you compute the composite health score by following the instructions in the Github repository. Then, repeat this survey on a regular cadence and discuss the results. Ideally, in a forum accessible by all members of the DAO.

We recognize some DAOs may want to use only part of the survey. If you choose to do this, apply the same formula in the repository to each independent factor by reviewing the associated questions as outlined in the file.

For further guidance on how to analyze your results, join us on the talentDAO Discord. We will be happy to answer any questions.

talentDAO and the future of the DAO Health initiative

This overall work represents the first part of a larger research initiative to develop a scientifically validated DAO health survey. Such a process [as is the way of science] takes time.

The survey is meant to be open-source for all DAOs to utilize. However, there will be future iterations of this survey that will be released as we continue our effort to produce a scientifically validated survey by conducting several validation studies over the next few years. Once a scientifically validated survey has been achieved, talentDAO will encourage participation from the entire DAO ecosystem to submit their results for a comprehensive annual report on DAO Health.

Further, along with the report talentDAO intends to list the anonymized data on the Ocean marketplace, where royalties will be distributed to participants for any sales.

It is important to note that this first iteration of the DAO Health survey is an empirically-driven pre-validated survey. That means that the research methodology necessary to confidently claim this survey measures what we say it does has not yet been fully conducted. As we conduct the research necessary to demonstrate scientific validity, we will release updated versions of the survey if we have confidence in a more precise measure.

That said, given that the context that these measures have been adapted from is not completely different — both DAOs and traditional organizations are human organizations — we have a good reason to believe that the validity of these measures should still hold.

As an open-source tool, DAO leaders can access it free forever. While it can be modified for your own DAO, we recommend consulting with one of our organizational science experts to help you make the best decision.

talentDAO intends to continue conducting research and providing scientifically-backed tools and insights to the DAO ecosystem.

Footnotes

1. Harter, James, Schmidt, Frank. 2002. Business-Unit-Level Relationship Between Employee Satisfaction, Employee Engagement, and Business Outcomes: A Meta-Analysis. Journal of Applied Psychology.
2. Gagnon, Chris, John, Elizabeth and Theunissen, Rob. Organizational Health a fastrack to performance improvement. Mckinsey & Company.
3. Buterin, Vitalik, 2014. DAOs, DACs, DAs and More: An Incomplete Terminology Guide
4. Daft, Richard L., 1983. Organizational Theory & Design
5. Guzzo, R.A., & Dickson, M.W. (1996). Teams in organizations: recent research on performance and effectiveness. Annual review of psychology, 47, 307-38.
6. Bell, Bradford & Kozlowski, Steve, 2002. A Typology of Virtual Teams: Implications for Effective Leadership. Faculty Publications - Human Resource Studies.
7. The expert review panel adapted the Delphi method for the item reduction process.
8. We did not operationalize the core team as it was not a prerequisite to operationalizing DAO health, however, we view core teams as both contributors themselves and facilitators of contributions from others. Typically, core team membership is explicitly assigned so there is little ambiguity about it.

A comprehensive list of references and reviewed literature can be found in the Github repository for this project.

This work was authored in a collaborative effort by organizational scientists at talentDAO: k3nn.eth, Nemo, Mr. Nobody, Sherifoz.

About talentDAO — talentDAO envisions a future where 1 billion people have access to ‘decent work’ through self-sovereign work arrangements offered by thriving, successful DAOs. We leverage organizational science to conduct research, build products, and offer talent strategy consulting services to the DAO ecosystem.