[吴恩达团队自然语言处理第二课_3]词嵌入与神经网络

[吴恩达团队自然语言处理第二课_3]词嵌入与神经网络

Overview

主要应用

  • Semantic analogies and similarity
  • Sentiment analysis
  • Classification of
    customer feedback

高级应用

  • Machine translation
  • Information extraction
  • Question answering

Learning objectives

  • Identify the key concepts of word representations
  • Generate word embeddings
  • Prepare text for machine learning
  • Implement the continuous bag-of-words model

Basic word representation

Integers

给词汇表每个单词分配一个唯一整数

  • 简单

  • Nunber按字母顺序排列，没有什么语义，如happy的值比hand大没有原因

One-hot vectors

  • 简单

  • No implied ordering不暗示两个词有任何关系

  • Huge vectors可能超过一百万行

  • No embedded meaning 不包含单词意思，如无法计算相似度，happy应该更接近excited

Word embeddings

Meaning as vectors

可以发现happpy与excited更接近而不是paper

获得了一些语义但是也放弃了一些精确度，精确度不如one-hot，因为在此2D图中，两个单词可能位于同一点,如spider和snake

word embeddings

  • 维数低

  • Embed meaning 有语义

    • e.g. semantic distance

      forest ≈tree forest≠ticket

    • e.g. analogies类比

      Paris:France::Rome:?

Terminology

Summary

  • Words as integers
  • Words as vectors.
    • One-hot vectors
    • Word embedding vectors
  • Benefits of word embeddings for NLP

How to create word embedding

Methods

Basic word embedding methods

  • word2vec(Google,2013)
    • Continuous bag-of-words(CBOW) 根据给定的单词预测丢失的单词
    • Continuous skip-gram/Skip-gram with negative sampling(SGNS) 根据给定的词预测词
  • Global Vectors(GloVe)(Stanford,2014)
  • fast Text(Facebook,2016)
    • Supports out-of-vocabulary (OOV) words

Advanced word embedding methods

Deep learning, contextual embeddings 单词根据context有不同的embedding，增加了多义词或相似词

  • BERT(Google,2018)
  • ELMo (Allen Institute for Al,2018)
  • GPT-2(OpenAl,2018)

以上都 Tunable pre-trained models available

Continuous bag-of-words(CBOW) model

基于周围单词得缺失单词

如果两个词周围的单词很像，那么他们可能语义相关

Create a training example

C是模型超参，可改，将中心词加上上下文词称为window，此处window=1+2+2=5

From corpus to training

e.g. 刚开始window内容为I am happy because I,然后window像右边滑动得am happy I am.

如此重复获得训练示例

CBOW in a nutshell

Cleaning and tokenization

Example in Python: corpus

# pip install nltk
# pip instalL emoji

import nltk
from nltk.tokenize import word_tokenize
import emoji
import re

nltk.download('punkt')  # download pre-trained Punkt tokenizer for English
corpus = 'Who ?"word embeddings" in 2020? I do!!!'
data = re.sub(r'[,!2;-]+','.', corpus)
data = nltk.word_tokenize(data)# tokenize string to words
data = [ ch.lower() for ch in data
         if ch. isalpha()
         or ch == '.'
         or emoji.get_emoji_regexp().search(ch)
         ]
print(data)
#['who', '?', 'word', 'embeddings', 'in', 'i', 'do', '.']

#Sliding window of words
def get_windows(words,C):
    i=C
    while i
for x,y in get_windows(
			['i','am','happy','because','i','am','learning'],
    		2
		):
	print(f'{x}\t{y}')
#['i', 'am', 'because', 'i']	happy
#['am', 'happy', 'i', 'am']	because
#['happy', 'because', 'am', 'learning']	i

Transforming words into vectors

Average of individual one-hot vectors

Final prpared training set

vector都是行向量，这样写是为了好看

Architecture of the CBOW model

Demension(single input)

行列向量 转置

Dimensions (batch input)

m : batch size 超参

组合成一个 V 行m 列的矩阵 称为 X

将y_hat分解为m个列向量

Rectified Linear Unit(ReLU)

Softmax

e.g.

Training a CBOW model

Loss

Corss-entropy loss

常在分类中使用

\[J=-\sum_{k=1}^V y_k\log \hat y_k \]

\[J=-\log \hat y_{actual\ word} \]

forward propagation

Training process

  • forward propagation
  • Cost
  • Backpropagation and gradient descent

forward propagation

Cost

Cost: mean of losses

\[J_{batch}=-\frac{1}{m}\sum_{i=1}^m\sum_{j=1}^Vy_j^{(i) }\log \hat y_j^{(i)}\\ J_{batch}=-\frac{1}{m}\sum_{i=1}^mJ^{(i)} \]

Minimizing the cost

  • Backpropagation:calculate partial derivatives of cost with respect to
    weights and biases

    \[\frac{\partial J_{batch}}{\partial W_1}, \frac{\partial J_{batch}}{\partial W_2}, \frac{\partial J_{batch}}{\partial b_1}, \frac{\partial J_{batch}}{\partial b_2} \]

    \[J_{batch}=f(W_1,W_2,b_1,b_2) \]

  • Gradient descent:update weights and biases

Backpropagation

Gradient descent

Extracting word embedding vectors

Option 1

Option 2

Option 3

取前两个平均值

Evaluation word embedding

Intrinsic evaluation

Test relationships between words

  • Analogies

    • Semantic analogies
      “France"is to"Paris"as"Italy"is to<?>

    • Syntactic analogies
      “seen"is to"saw"as"been"is to<?>

    • Ambiguity
      “wolf”is to"pack"as"bee"is to <?>→swarm? colony?

    • 在化学元素对应做的不好，word2vec给出：

  • Clustering

  • Visualization

Extrinsic evaluation

Test word embeddings on external task

e.g. named entity recognition, parts-of-speech tagging

  • Evaluates actual usefulness of embeddings
  • 比内在评估更耗时
  • More difficult to troubleshoot

Conclusion

Recap and assignment

  • Data preparation
  • Word representations
  • Continuous bag-of-words model
  • Evaluation

Going further

  • Advanced language modelling and word embeddings
  • NLP and machine learning libraries
NLP

相关

NLP的前世今生

安装pyhanlp报错，之解决方案

NLP中的预训练语言模型（三）—— XL-Net和Transformer-XL

5个很少被提到但能提高NLP工作效率的Python库

中文分词工具探析（一）：ICTCLAS (NLPIR)

NLP（四十一）：解决样本不均衡FocalLoss与GHM

中文自然语言处理(NLP)(三)运用python jieba模块计算知识点当中关键词的词频

(转载)深入理解NLP Subword算法：BPE、WordPiece、ULM

NLP学习参考

Elasticsearch应用Hanlp中文分词和Pinyin 拼音分词

NLP面试整理

CBLUE 中文医疗信息处理基准 NLP (论文复现流程)

标签