LIME is a popular local-explanation technique, which seeks to build a linear classifier in the immediate area of a particular data input..
Links:

LIME creates local linear classifier at biundary between classes of a black-box model. Image ©2016 Marco Tulio Correia Ribeiro