Summary of the paper
| Title | A Lightweight and Efficient Tool for Cleaning Web Pages |
|---|---|
| Authors | Stefan Evert |
| Abstract | Originally conceived as a “naïve” baseline experiment using traditional n-gram language models as classifiers, the NCleaner system has turned out to be a fast and lightweight tool for cleaning Web pages with state-of-the-art accuracy (based on results from the CLEANEVAL competition held in 2007). Despite its simplicity, the algorithm achieves a significant improvement over the baseline (i.e. plain, uncleaned text dumps), trading off recall for substantially higher precision. NCleaner is available as an open-source software package. It is pre-configured for English Web pages, but can be adapted to other languages with minimal amounts of manually cleaned training data. Since NCleaner does not make use of HTML structure, it can also be applied to existing Web corpora that are only available in plain text format, with a minor loss in classfication accuracy. |
| Language | Language-independent |
| Topics | Tools, systems, applications, Language modelling, Tagging |
Full paper |
A Lightweight and Efficient Tool for Cleaning Web Pages |
Slides |
A Lightweight and Efficient Tool for Cleaning Web Pages |
| Bibtex | @InProceedings{EVERT08.885, author = {Stefan Evert}, title = {A Lightweight and Efficient Tool for Cleaning Web Pages}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, month = {may}, date = {28-30}, address = {Marrakech, Morocco}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-4-0}, note = {http://www.lrec-conf.org/proceedings/lrec2008/}, language = {english} } |