HOME / RESEARCH / CROWDSOURCING & OPEN INNOVATION / NASA TOURNAMENT LAB /

U.S. Patent and Trademark Office Algorithm Challenge

Historical patents are a difficult target for digitization. Figures and labels are central to patents and present complex combinations of image and text elements. Sponsored by the CoECI in 2011, the U.S. Patent and Trademark Office (USPTO) Algorithm Challenge tasked competitors to develop algorithms that automatically detect figures and part labels in US patents (USPTO; White House blog; Topcoder problem statement). The challenge drew 232 two-person teams, of which 70 teams (30%) submitted solutions. The first place submission was able to correctly identify part labels within an image and correctly recognize the text in more than 70% of the test cases. For more information see Riedl et al. (2014).

See also: Creativity & Problem-Solving, Crowdsourcing & Open Innovation, Data Science & AI Development, Project, Contest, Image Processing, Machine Learning Algorithms
  • Christoph Riedl, Richard Zanibbi, Marti A. Hearst, Siyu Zhu, Michael Menietti, Jason Crusan, Ivan Metelsky, and Karim R. Lakhani. 2016. “Detecting Figures and Part Labels in Patents: Competition-Based Development of Image Processing Algorithms.” International Journal on Document Analysis and Recognition (IJDAR), 19, 2, Pp. 155-172. Publisher's VersionAbstract
    Detecting_Figures_and_Part_Labels_in_Patents.pdf

    Most United States Patent and Trademark Office (USPTO) patent documents contain drawing pages which describe inventions graphically. By convention and by rule, these drawings contain figures and parts that are annotated with numbered labels but not with text. As a result, readers must scan the document to find the description of a given part label. To make progress toward automatic creation of ‘tool-tips’ and hyperlinks from part labels to their associated descriptions, the USPTO hosted a monthlong online competition in which participants developed algorithms to detect figures and diagram part labels. The challenge drew 232 teams of two, of which 70 teams (30 %) submitted solutions. An unusual feature was that each patent was represented by a 300-dpi page scan along with an HTML file containing patent text, allowing integration of text processing and graphics recognition in participant algorithms. The design and performance of the top-5 systems are presented along with a system developed after the competition, illustrating that the winning teams produced near state-of-the-art results under strict time and computation constraints. The first place system used the provided HTML text, obtaining a harmonic mean of recall and precision (F-measure) of 88.57 % for figure region detection, 78.81 % for figure regions with correctly recognized figure titles, and 70.98 % for part label detection and recognition. Data and source code for the top-5 systems are available through the online UCI Machine Learning Repository to support follow-on work by others in the document recognition community.