Sketch Search

Sketch Interfaces

Sketch Search is a system developed during my Research Experience for Undergraduates at University of North Carolina at Charlotte. I joined an ongoing project with Dr. Richard Souvenir’s research group: tackling the problem of content-based video retrieval (CBVR) using sketching. CBVR is an alternative to traditional video search, such as searching for videos on YouTube, which require the videos to be manually annotated for keyword matching. By matching videos based on distinguishing features, it is possible to search without relying on the sometimes inaccurate or ambiguous text within video descriptions. Sketching is one input method for video search.

Our system, Sketch Search, is targeted towards human action. It prompts the user to draw a sketch of a human figure with motion cues.  It then animates the user sketch for use as a query video. A previous graduate student, Evan Suma, created two simple sketch interfaces and tested search results using a feature descriptor called the R-transform. The initial results showed that silhouette based feature descriptors like R-transform worked well for comparing sketches and video, but many questions about usability remained. I was tasked with creating another sketch interface and polishing the user interface in preparation for a user study. Along the way, I brushed up on the research background by reading and summarizing research papers.  I developed a detailed understanding of the algorithms behind the search procedure that later aided me in developing heuristics to analyze the quality of the searches.

The following summer, I organized and ran a user study (with 81 participants) to evaluate each interface and measure the accuracy of search results. I used the collected sketches to build a database for testing alternative feature descriptors. I then developed a new video descriptor by combining tree-structured vector quantization with an existing frame descriptor and ran initial analysis on the results of the user study. Preliminary results were presented at the Charlotte Summer Research Symposia, and I won first prize at the UNC Charlotte Computer Science REU Site Project Presentations. Recently, we submitted a research paper titled,  “Evaluating Query Interfaces for Articulated Motion Video Search,” on the sketch interfaces and user study results to the International Journal of Human-Computer Studies.