Welcome to the Wiki of the seminar ''Deep Natural Language Processing'' in the winter semester 2020/2021

This seminar is organized at the chair of Prof. Dr. Hannah Bast by Theresa Klumpp, Matthias Hertel and Natalie Prange. The seminar will take place every Tuesday, 2:15 pm - 3:45 pm, in the seminar room SR 00-010/14 in building 101 (if the COVID-19 conditions allow it) and via Zoom for those of you who want to attend online (see below for link and password). Attendance in one of these two forms is compulsory. There will be no session on Tuesday, November 10th, 2020 (seminar places are assigned by the HISinOne in that week) and on Tuesday, December 29th, 2020 + Tuesday, January 5th, 2021 (Christmas break).

Important Links

• Please register for the course in HISinOne and Daphne.

• There is an introduction to SVN in English and German.

• There is a forum for important announcements and questions you might have. Please check the forum regularly!

• The Zoom meeting we will use throughout the seminar: https://uni-freiburg.zoom.us/j/87399563594 (Password: DeepNLP20)

Modalities

Participants of the seminar will have to present one of the topics either alone or as a group of two. Each presentation will be 30 minutes for one participant or 2 * 20 minutes for two. In addition to introducing the topic each presentation must include a demo part where participants present a practical application of their topic.

What exactly this demo entails depends on the topic and will be discussed with each person/team separately. While we will provide suggestions you are very welcome to bring in your own ideas. Examples for demos may include the implementation of a small application, an interactive visualization or the demonstration of a complex existing system which you have set up on your own.

Schedule for each individual presentation

• Before Meeting 1: Research the given topic (starting from the pointers given below) and make a plan of what you want to talk about

• Meeting 1 (3 weeks before your presentation): show us your plan + we settle on the scope of your presentation

• Before Meeting 2: Understand / work out all the necessary details and play around (extensively) with existing software or write your own

• Meeting 2 (2 weeks before your presentation): show us what you have done + we try to help with remaining problems

• Before Meeting 3: Prepare your presentation and the demos you want to show

• Meeting 3 (1 week before your presentation): show us what you have prepared + we help with remaining problems

• Before your presentation: Finish your presentation and demo, including all the details

Sessions

Topics

The topics are going to be introduced and roughly explained in the first session. They are basically about how Deep Learning can be used in Natural Language Processing.

Please note that you are supposed to present the topic, not the material listed here. The material is only intended as a starting point for your research.

• Standard Language Model

  • Having a model for natural language is the base for many NLP tasks. N-gram models are a simple way to obtain such a model.

  • Stanford lecture notes

  • Stanford lecture

  • NLTK Python toolkit for implementation

• RNN Language Model

  • Recurrent Neural Network (RNN) language models are in general superior to n-gram language models because they can model long-term dependencies. Explain RNNs, LSTM Networks and how they are used to model language.

  • MIT lecture

  • Blog post

• word2vec

  • Word2vec is a technique to represent words as vectors from a high dimensional vector space. The goal is that words that are semantically similar have similar vectors. This is a central method in many NLP problems.

  • The original paper that introduced Word2Vec in 2013

  • Stanford lecture

  • Python framework for word vector generation

• Attention

  • With the attention mechanism, a neural network can learn to focus on specific parts of the input. This has applications in Machine Translation, Language Modeling, Image Captioning and many more.

  • Jointly learning to align and translate

  • Effective approaches to attention

  • Application to Image Captioning: Paper 1 and Paper 2

• Transformer models

  • A new neural network architecture achieving state-of-the-art results in many NLP tasks. It is used by OpenAI in their famous GPT-2 paper to automatically generate text that is almost indistinguishable from human-written text.

  • Attention is all you need

  • Generative Pre-Training (GPT-2)

• Bidirectional Encoder Representations from Transformers (BERT)

  • BERT is a method for NLP pre-training based on the Transformer architecture. It is successfully applied in a large variety of NLP tasks.

  • Original paper

  • Blog post

  • Blog post

• Convolutional Neural Networks for NLP

  • While originally stemming from Image Analysis, Convolutional Neural Networks also have their applications in Natural Language Processing.

  • Comparison of CNN and RNN for NLP

  • Convolutional Sequence to Sequence Learning

• Text Classification

  • The goal is to classify text using Machine Learning. Examples of NLP-Applications are sentiment analysis, topic labeling or spam detection.

  • Survey that covers many algorithms and methods on text classification

  • Hands-on guide to implementation

• Sentiment Analysis

  • The task of identifying and analyzing opinions about entities and their aspects in text.

  • Survey paper

  • Popular but not so recent paper

• Question Answering on Text

  • The task of extracting an answer to a given question from a given document/paragraph (reading comprehension) or a large set of documents like Wikipedia (open domain question answering). The most prominent dataset for reading comprehension tasks is currently the SQuAD dataset.

  • Stanford lecture

  • SQuAD explorer

• Question Answering on Knowledge Bases

  • The task of extracting an answer to a given questions from a knowledge base like Wikidata.

  • Microsoft Research QA system STAGG

  • Our QA system Aqqu (code on request)

  • Try out Aqqu

• Machine Translation

  • In the last years, machine translation systems like Google Translate and DeepL have made big progress. We will look at such a system in detail, and see how it is even possible to translate between language pairs the model has never seen during training.

  • Google Translate

  • Multilingual Translation

  • Is machine translation better than humans? Blogpost 1 and Blogpost 2

• Bias in Language Models

  • Language Models can only be as good as the input we give them (“Garbage in, garbage out”). If the input data is biased, the models will mimic that behavior. This can lead to real life problems.

  • Identifying (different kinds of) bias

  • This literature review gives a good overview of gender bias in different areas of NLP and contains lots of other papers and resources.

• Named Entity Disambiguation

  • Named Entity Disambiguation, also referred to as entity linking, is the task of linking named entities in text to their corresponding entries in a knowledgebase like Wikidata or Wikipedia.

  • Ambiverse Entity Disambiguation Demo

  • Ambiverse Entity Disambiguation paper

• Reinforcement Learning

  • RL has many applications in NLP. You should pick one or two that you are interested in and focus on them.

  • Examples: Text-based games, Dialogue Generation or Question Answering

• Automatic Hyperparameter Optimization

  • When designing and training neural networks, many decisions have to be taken about the network architecture and training process, which affect the final outcome. Manually tuning these decisions is a tedious task. Recent work automates the process of finding the optimal setting.

  • A study of the impact of various hyperparameters on a text classification task

  • BOHB: Bayesian Optimization and Hyperband

• More possible topics

  • Speech Recognition (a.k.a. Speech to text)
  • Text Summarization

  • Chat bots & dialogues

  • Parsing & semantic role labeling

  • Transfer Learning and Multi-task learning with the Text-to-text transfer transformer (T5): Blogpost and Paper

  • Personality Detection from Text

  • Fake News Detection