Deja Vu: Characterizing Worker Reliability Using Task Consistency

Abstract:

Consistency is a practical metric that evaluates an instrument’s reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker’s reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task — such as placement, difficulty, and transformation — affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.