Building Web Reputation Systems

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Book description

What do Amazon's product reviews, eBay's feedback score system, Slashdot's Karma System, and Xbox Live's Achievements have in common? They're all examples of successful reputation systems that enable consumer websites to manage and present user contributions most effectively. This book shows you how to design and develop reputation systems for your own sites or web applications, written by experts who have designed web communities for Yahoo! and other prominent sites.

Building Web Reputation Systems helps you ask the hard questions about these underlying mechanisms, and why they're critical for any organization that draws from or depends on user-generated content. It's a must-have for system architects, product managers, community support staff, and UI designers.

• Scale your reputation system to handle an overwhelming inflow of user contributions
• Determine the quality of contributions, and learn why some are more useful than others
• Become familiar with different models that encourage first-class contributions
• Discover tricks of moderation and how to stamp out the worst contributions quickly and efficiently
• Engage contributors and reward them in a way that gets them to return
• Examine a case study based on actual reputation deployments at industry-leading social sites, including Yahoo!, Flickr, and eBay

Publisher resources

Table of contents

1. Preface
   1. What Is This Book About?
   2. Why Write a Book About Reputation?
   3. Who Should Read This Book
   4. Organization of This Book
      1. Part I: Reputation Defined and Illustrated
      2. Part II: Extended Elements and Applied Examples
      3. Part III: Building Web Reputation Systems
      4. Role-Based Reading (for Those in a Hurry)
      1. From Randy
      2. From Bryce
      1. 1. Reputation Systems Are Everywhere
         1. An Opinionated Conversation
         2. People Have Reputations, but So Do Things
         3. Reputation Takes Place Within a Context
         4. We Use Reputation to Make Better Decisions
         5. The Reputation Statement
            1. Explicit: Talk the Talk
            2. Implicit: Walk the Walk
            3. The Minimum Reputation Statement
            1. Local Reputation: It Takes a Village
            2. Global Reputation: Collective Intelligence
            3. FICO: A Study in Global Reputation and Its Challenges
            4. Web FICO?
            1. Attention Doesn’t Scale
            2. There’s a Whole Lotta Crap Out There
            3. People Are Good. Basically.
               1. Know thy user
               2. Honor creators, synthesizers, and consumers
               3. Throw the bums out
               1. The Reputation Statement and Its Components
                  1. Reputation Sources: Who or What Is Making a Claim?
                  2. Reputation Claims: What Is the Target’s Value to the Source? On What Scale?
                  3. Reputation Targets: What (or Who) Is the Focus of a Claim?
                  1. Messages and Processes
                  2. Reputation Model Explained: Vote to Promote
                  3. Building on the Simplest Model
                  1. From Reputation Grammar to…
                  1. 3. Building Blocks and Reputation Tips
                     1. Extending the Grammar: Building Blocks
                        1. The Data: Claim Types
                           1. Qualitative claim types
                              1. Text comments
                              2. Media uploads
                              3. Relevant external objects
                              1. Normalized value
                              2. Rank value
                              3. Scalar value
                              1. Roll-ups: Counters, accumulators, averages, mixers, and ratios
                                 1. Simple Counter
                                 2. Reversible Counter
                                 3. Simple Accumulator
                                 4. Reversible Accumulator
                                 5. Simple Average
                                 6. Reversible Average
                                 7. Mixer
                                 8. Simple Ratio
                                 9. Reversible Ratio
                                 1. Simple normalization (and weighted transform)
                                 2. Scalar denormalization
                                 3. External data transform
                                 1. Common decision process patterns
                                    1. Simple Terminator
                                    2. Simple Evaluator
                                    3. Terminating Evaluator
                                    4. Message Splitter
                                    5. Conjoint Message Delivery
                                    1. Typical inputs
                                    2. Reputation statements as input
                                    3. Periodic inputs
                                    1. Return values
                                    2. Signals: Breaking out of the reputation framework
                                    3. Logging
                                    1. The Power and Costs of Normalization
                                    2. Liquidity: You Won’t Get Enough Input
                                    3. Bias, Freshness, and Decay
                                       1. Ratings bias effects
                                       2. First-mover effects
                                       3. Freshness and decay
                                       1. Simple Models
                                          1. Favorites and Flags
                                             1. Vote to promote
                                             2. Favorites
                                             3. Report abuse
                                             1. Participation karma
                                             2. Quality karma
                                             3. Robust karma
                                             1. User Reviews with Karma
                                             2. eBay Seller Feedback Karma
                                             3. Flickr Interestingness Scores for Content Quality
                                             1. Party Crashers
                                             2. Keep Your Barn Door Closed (but Expect Peeking)
                                                1. Decay and delay
                                                2. Provide a moving target
                                                1. 5. Planning Your System’s Design
                                                   1. Asking the Right Questions
                                                      1. What Are Your Goals?
                                                         1. User engagement
                                                         2. Establishing loyalty
                                                         3. Coaxing out shy advertisers
                                                         4. Improving content quality
                                                         1. Web 1.0: Staff creates, evaluates, and removes
                                                         2. Bug report: Staff creates and evaluates, users remove
                                                         3. Reviews: Staff creates and removes, users evaluate
                                                         4. Surveys: Staff creates, users evaluate and remove
                                                         5. Submit-publish: Users create, staff evaluates and removes
                                                         6. Agents: Users create and remove, staff evaluates
                                                         7. Basic social media: Users create and evaluate, staff removes
                                                         8. The Full Monty: Users create, evaluate, and remove
                                                         1. Predictably irrational
                                                         2. Incentives and reputation
                                                         3. Altruistic or sharing incentives
                                                            1. Tit-for-tat and pay-it-forward incentives
                                                            2. Friendship incentives
                                                            3. Crusader, opinionated incentives, and know-it-all
                                                            1. Direct revenue incentives
                                                            2. Incentives through branding: Professional promotion
                                                            1. Fulfillment incentives
                                                            2. Recognition incentives
                                                            3. Personal or private incentives: The quest for mastery
                                                            1. What are people there to do?
                                                            2. Is this a new community? Or an established one?
                                                            3. The competitive spectrum
                                                            1. The Objects in Your System
                                                               1. Architect, Understand Thyself
                                                                  1. So…what does your application do?
                                                                  2. Perform an application audit
                                                                  1. People are interested in it
                                                                  2. The decision investment is high
                                                                  3. The entity has some intrinsic value worth enhancing
                                                                  4. The entity should persist for some length of time
                                                                  1. User Actions Make Good Inputs
                                                                     1. Explicit claims
                                                                     2. Implicit claims
                                                                     1. Emphasize quality, not simple activity
                                                                     2. Rate the thing, not the person
                                                                     3. Reward firsts, but not repetition
                                                                     4. Use the right scale for the job
                                                                     5. Match user expectations
                                                                     1. The ratings life cycle
                                                                        1. The interface design of reputation inputs
                                                                        1. The schizophrenic nature of stars
                                                                        2. Do I like you, or do I “like” like you
                                                                        1. Favorites, forwarding, and adding to a collection
                                                                           1. Favorites
                                                                           2. Forwarding
                                                                           3. Adding to a collection
                                                                           1. Context Is King
                                                                           2. Limit Scope: The Rule of Email
                                                                           3. Applying Scope to Yahoo! EuroSport Message Board Reputation
                                                                           1. The Heart of the Machine: Reputation Does Not Stand Alone
                                                                           2. Common Reputation Generation Mechanisms and Patterns
                                                                              1. Generating personalization reputation
                                                                              2. Generating aggregated community ratings
                                                                                 1. Ranking large target sets (preference orders)
                                                                                 1. Points as currency
                                                                                 1. How to Use a Reputation: Three Questions
                                                                                 2. Who Will See a Reputation?
                                                                                    1. To Show or Not to Show?
                                                                                    2. Personal Reputations: For the Owner’s Eyes Only
                                                                                    3. Personal and Public Reputations Combined
                                                                                    4. Public Reputations: Widely Visible
                                                                                    5. Corporate Reputations Are Internal Use Only: Keep Them Hush-hush
                                                                                    1. Reputation Filtering
                                                                                    2. Reputation Ranking and Sorting
                                                                                    3. Reputation Decisions
                                                                                    1. Content Reputation
                                                                                    2. Karma
                                                                                       1. Karma is complex, built of indirect inputs
                                                                                       2. Karma calculations are often opaque
                                                                                       3. Display karma sparingly
                                                                                       4. Karma caveats
                                                                                       1. Normalized Score to Percentage
                                                                                       2. Points and Accumulators
                                                                                       3. Statistical Evidence
                                                                                       4. Levels
                                                                                          1. Numbered levels
                                                                                          2. Named levels
                                                                                          1. Leaderboard ranking
                                                                                          2. Top-X ranking
                                                                                          1. Leaderboards Considered Harmful
                                                                                             1. What do you measure?
                                                                                             2. Whatever you do measure will be taken way too seriously
                                                                                             3. If it looks like a leaderboard and quacks like a leaderboard…
                                                                                             4. Leaderboards are powerful and capricious
                                                                                             5. Who benefits?
                                                                                             1. Up with the Good
                                                                                                1. Rank-Order Items in Lists and Search Results
                                                                                                2. Content Showcases
                                                                                                   1. The human touch
                                                                                                   1. Configurable Quality Thresholds
                                                                                                   2. Expressing Dissatisfaction
                                                                                                   1. Reporting Abuse
                                                                                                      1. Who watches the watchers?
                                                                                                      1. Inferred Reputation for Content Submissions
                                                                                                         1. Just-in-time reputation calculation
                                                                                                         1. On the User Profile
                                                                                                            1. My Affiliations
                                                                                                            2. My History
                                                                                                            3. My Achievements
                                                                                                            1. Integrating with Your Application
                                                                                                               1. Implementing Your Reputation Model
                                                                                                               2. Rigging Inputs
                                                                                                               3. Applied Outputs
                                                                                                               4. Beware Feedback Loops!
                                                                                                               5. Plan for Change
                                                                                                               1. Bench Testing Reputation Models
                                                                                                               2. Environmental (Alpha) Testing Reputation Models
                                                                                                               3. Predeployment (Beta) Testing Reputation Models
                                                                                                                  1. Performance: Testing scale
                                                                                                                  2. Confidence: Testing computation accuracy
                                                                                                                  3. Application optimization: Measuring use patterns
                                                                                                                  4. Feedback: Evaluating customer’s satisfaction
                                                                                                                  5. Value: Measuring ROI
                                                                                                                  1. Tuning for ROI: Metrics
                                                                                                                     1. Model tuning
                                                                                                                     2. Application tuning
                                                                                                                     1. Emergent effects and emergent defects
                                                                                                                        1. Defending against emergent defects
                                                                                                                        1. What Is Yahoo! Answers?
                                                                                                                           1. A Marketplace for Questions and Yahoo! Answers
                                                                                                                           2. Attack of the Trolls
                                                                                                                              1. Time was a factor
                                                                                                                              2. Location, location, location
                                                                                                                              1. Setting Goals
                                                                                                                                 1. Cutting costs
                                                                                                                                 2. Cleaning up the neighborhood
                                                                                                                                 1. The Objects
                                                                                                                                 2. Limiting Scope
                                                                                                                                 3. An Evolving Model
                                                                                                                                    1. Iteration 1: Abuse reporting
                                                                                                                                       1. Inputs
                                                                                                                                       2. Mechanism and diagram
                                                                                                                                       3. Analysis
                                                                                                                                       1. Inputs
                                                                                                                                       2. Mechanism and diagram
                                                                                                                                       3. Analysis
                                                                                                                                       1. Inputs
                                                                                                                                       2. Mechanism and diagram
                                                                                                                                       3. Analysis
                                                                                                                                       1. Inputs
                                                                                                                                       2. Mechanism and diagram
                                                                                                                                       3. Analysis
                                                                                                                                       1. Who Will See the Reputation?
                                                                                                                                       2. How Will the Reputation Be Used to Modify Your Site’s Output?
                                                                                                                                       3. Is This Reputation for a Content Item or a Person?
                                                                                                                                       1. Application Integration
                                                                                                                                       2. Testing Is Harder Than You Think
                                                                                                                                       3. Lessons in Tuning: Users Protecting Their Power
                                                                                                                                       1. Reputation Framework Requirements
                                                                                                                                          1. Calculations: Static Versus Dynamic
                                                                                                                                             1. Static: Performance, performance, performance
                                                                                                                                             2. Dynamic: Reputation within social networks
                                                                                                                                             1. Performance at scale
                                                                                                                                             1. The Invisible Reputation Framework: Fast, Cheap, and Out of Control
                                                                                                                                                1. Requirements
                                                                                                                                                2. Implementation details
                                                                                                                                                3. Lessons learned
                                                                                                                                                1. Yahoo! requirements
                                                                                                                                                2. Yahoo! implementation details
                                                                                                                                                   1. High-level architecture
                                                                                                                                                   2. Messaging dispatcher
                                                                                                                                                   3. Model execution engine
                                                                                                                                                   4. External signaling interface
                                                                                                                                                   5. Reputation repository
                                                                                                                                                   6. Reputation query interface
                                                                                                                                                   1. Recommendations for All Reputation Frameworks
                                                                                                                                                   1. Further Reading
                                                                                                                                                   2. Recommender Systems
                                                                                                                                                   3. Social Incentives
                                                                                                                                                   4. Patents
                                                                                                                                                   Show and hide more

                                                                                                                                                   Product information

                                                                                                                                                   • Title: Building Web Reputation Systems
                                                                                                                                                   • Author(s): Bryce Glass, F. Randall Farmer
                                                                                                                                                   • Release date: March 2010
                                                                                                                                                   • Publisher(s): O'Reilly Media, Inc.
                                                                                                                                                   • ISBN: 9780596159795