New to Dream to Learn? Check out our Quick Start guide!  

0COMMENTS0RECOMMENDS

Ground Truth Analysis Tool for Retrieve and Rank

11
POSTED IN: Building Bridges from R to IBM Watson

IBM Watson: Retrieve & Rank Ground Truth Analysis Tool

 

•RANKER: To return the most relevant documents at the top of your results, the Retrieve and Rank services uses a machine learning component called a ranker. You send queries to the trained ranker.
•GROUND TRUTH - The ranker learns from examples before it can re-rank results from queries that it hasn't seen before. Collectively, the examples are referred to as "ground truth.”
•Background
HYPOTHESIS: That the SHAPE and QUALITY of the Ground_Truth.CSV file for the Retrieve and Rank service, has attributes that can be measured – and impact Precision and Recall KPIs.
 
 

BENEFIT:  If we can create a tool to analyze a Ground_Truth.CSV file that is used for Retrieve and Rank, we can get a better understanding of performance - we can create a yardstick to help measure KPIs - GT attributes <-> Precision and Recall

Code

https://github.com/rustyoldrake/R_Scripts_for_Watson/blob/master/Watson_R%26R_Ground_Truth_CSV_Analysis_Tool.R

Deck

https://drive.google.com/file/d/0BwjxYjWyopXhNmtZY1o0TFVtcjQ/view?usp=sharing

Summary

•Exploration of simple tool, written in “R” can provide a 60 second analysis of GT file
•BENCHMARK GT files
–What’s ‘normal’ relative to other GT files
–How does GT “SHAPE” impact Precision and Recall
–KPIs – Measurable GT attributes that can be used to help assess overall performance of system – compare and contrast
•Ground truth ‘fingerprint’ 
•Get a better feel for shape and quality of GT used in retrieve and rank
 
 
 

Before you can comment, you need to sign-up or login

About the Author

Ryan Anderson

Ryan Anderson

Hi! I like to play with data, analytics and hack around with robots and gadgets in my garage. Lately I've been learning about machine learning.

About this blog

This is an informal blog that explores tools, code and tricks that group members have developed to engage IBM Watson cognitive computing services - from the R Programming Language. Packages include RCURL to access Watson APIs - for services that include Natural Language Classifier and Speech to Text. THIS IS MY PERSONAL BLOG - it does not represent the views of my employer. Code is presented as 'use at your own risk' (it has lots of bugs)

Created: September 13, 2015

English

Up Next