Overview of Word Embeddings
Word embeddings, in short, are numerical representations of text. They are represented as ‘n-dimensional’ vectors where the number of dimensions ‘n’ is determined on the corpus size and the expressiveness desired. The larger the size of your corpus, the larger you want ‘n’. A larger ‘n’ also allows you to capture more features in the embedding. However, a larger dimension involves a longer and more difficult optimization process so a sufficiently large ‘n’ is what you want to use, determining this size is often problem-specific. Since a lot of work has been done in the research community, one could use the dimension hyperparameters used for similar problems as a baseline.
Word embeddings embed meaning of text in a vector space. What this means is that words that are closer in meaning, i.e. synonyms, have near identical vector representations. In other words, words that are similar in meaning have low distance in the high-dimensional vector space and words that are unrelated have high distance. Mikolov came up with a way to train a shallow network to create a vector representation. The generation of these vectors is conceptually similar to how an autoencoder functions to create a low-dimensional representation of the input data.
This requires, as part of preprocessing, the generation of (context , target) pairs. Given a ‘context’, i.e. a certain number of continuous words in a sentence, predict the ‘target’, i.e. word/set of words that are associated with it in a sentence. An example of an association is a word that follows the context or precedes the context. It can also be a word in between n context words. Once this has been generated, it is passed to the Neural network to train. In scalable word-embedding-based NLP algorithms, optimizations such as negative sampling help to significantly speed up computation.
These are implementations of both the Continuous Bag of Words(CBOW) and Skipgram approaches. These do not have hierarchical softmax, negative sampling or subsampling of frequent words introduced by Mikolov making it easy to illustrate or experiment with the fundamental concepts. Tokenization of the corpus should also be done prior to the generation of embeddings but this is also left out here for simplicity. Word embeddings can also be used to identify topics by clustering the embeddings and selecting the words closest to the centers of the clusters.
Example
For a brief overview of these algorithms, please refer to Mikolov’s papers ‘Distributed Representations of Words and Phrases and their Compositionality’ and ‘Efficient Estimation of Word Representations in Vector Space’.
Consider the following text as our entire corpus:
‘Empathy for the poor may not come easily to people who never experienced it’
Vocabulary size V = 14 and vector dimension N = 10
For context size = 3 and target size = 1, we generate (context,target) pairs such as
[('Empathy','for','the'),'poor']
[('for','the','poor'),'may']
[('the','poor','may'),'not']
...
Continuous Bag of Words
The following is a Pytorch implementation of the CBOW algorithm.
# Author: Srijith Rajamohan based off the work by Robert Guthrie
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import urllib.request
from nltk.tokenize import RegexpTokenizer
from nltk.corpus import stopwords
from nltk import word_tokenize
import sklearn
from sklearn.cluster import KMeans
from sklearn.metrics.pairwise import euclidean_distances
torch.manual_seed(1)
CONTEXT_SIZE = 3
EMBEDDING_DIM = 10
test_sentence = """Empathy for the poor may not come easily to people who never experienced it. They may blame the victims and insist their predicament can be overcome through determination and hard work.
But they may not realize that extreme poverty can be psychologically and physically incapacitating — a perpetual cycle of bad diets, health care and education exacerbated by the shaming and self-fulfilling prophecies that define it in the public imagination.
Gordon Parks — perhaps more than any artist — saw poverty as “the most savage of all human afflictions” and realized the power of empathy to help us understand it. It was neither an abstract problem nor political symbol, but something he endured growing up destitute in rural Kansas and having spent years documenting poverty throughout the world, including the United States.
That sensitivity informed “Freedom’s Fearful Foe: Poverty,” his celebrated photo essay published in Life magazine in June 1961. He took readers into the lives of a Brazilian boy, Flavio da Silva, and his family, who lived in the ramshackle Catacumba favela in the hills outside Rio de Janeiro. These stark photographs are the subject of a new book, “Gordon Parks: The Flavio Story” (Steidl/The Gordon Parks Foundation), which accompanies a traveling exhibition co-organized by the Ryerson Image Centre in Toronto, where it opens this week, and the J. Paul Getty Museum. Edited with texts by the exhibition’s co-curators, Paul Roth and Amanda Maddox, the book also includes a recent interview with Mr. da Silva and essays by Beatriz Jaguaribe, Maria Alice Rezende de Carvalho and Sérgio Burgi.
""".split()
# we should tokenize the input, but we will ignore that for now
# build a list of tuples. Each tuple is ([ word_i-2, word_i-1 ], target word)
def get_key(word_id):
for key,val in word_to_ix.items():
if(val == word_id):
print(key)
def cluster_embeddings(filename,nclusters):
X = np.load(filename)
kmeans = KMeans(n_clusters=nclusters, random_state=0).fit(X)
center = kmeans.cluster_centers_
distances = euclidean_distances(X,center)
for i in np.arange(0,distances.shape[1]):
word_id = np.argmin(distances[:,i])
print(word_id)
get_key(word_id)
def read_data(file_path):
tokenizer = RegexpTokenizer(r'\w+')
data = urllib.request.urlopen(file_path)
data = data.read().decode('utf8')
tokenized_data = word_tokenize(data)
stop_words = set(stopwords.words('english'))
stop_words.update(['.',',',':',';','(',')','#','--','...','"'])
cleaned_words = [ i for i in tokenized_data if i not in stop_words ]
return(cleaned_words)
#test_sentence = read_data('https://www.gutenberg.org/files/57884/57884-0.txt')
ngrams = []
for i in range(len(test_sentence) - CONTEXT_SIZE):
tup = [test_sentence[j] for j in np.arange(i , i + CONTEXT_SIZE) ]
ngrams.append((tup,test_sentence[i + CONTEXT_SIZE]))
# print the first 3, just so you can see what they look like
#print(ngrams)
vocab = set(test_sentence)
print("Length of vocabulary",len(vocab))
word_to_ix = {word: i for i, word in enumerate(vocab)}
class CBOWModeler(nn.Module):
def __init__(self, vocab_size, embedding_dim, context_size):
super(CBOWModeler, self).__init__()
self.embeddings = nn.Embedding(vocab_size, embedding_dim)
self.linear1 = nn.Linear(context_size * embedding_dim, 128)
self.linear2 = nn.Linear(128, vocab_size)
def forward(self, inputs):
embeds = self.embeddings(inputs).view((1, -1)) # -1 implies size inferred for that index from the size of the data
#print(np.mean(np.mean(self.linear2.weight.data.numpy())))
out1 = F.relu(self.linear1(embeds)) # output of first layer
out2 = self.linear2(out1) # output of second layer
#print(embeds)
log_probs = F.log_softmax(out2, dim=1)
return log_probs
def predict(self,input):
context_idxs = torch.tensor([word_to_ix[w] for w in input], dtype=torch.long)
res = self.forward(context_idxs)
res_arg = torch.argmax(res)
res_val, res_ind = res.sort(descending=True)
res_val = res_val[0][:3]
res_ind = res_ind[0][:3]
#print(res_val)
#print(res_ind)
for arg in zip(res_val,res_ind):
#print(arg)
print([(key,val,arg[0]) for key,val in word_to_ix.items() if val == arg[1]])
def freeze_layer(self,layer):
for name,child in model.named_children():
print(name,child)
if(name == layer):
for names,params in child.named_parameters():
print(names,params)
print(params.size())
params.requires_grad= False
def print_layer_parameters(self):
for name,child in model.named_children():
print(name,child)
for names,params in child.named_parameters():
print(names,params)
print(params.size())
def write_embedding_to_file(self,filename):
for i in self.embeddings.parameters():
weights = i.data.numpy()
np.save(filename,weights)
losses = []
loss_function = nn.NLLLoss()
model = CBOWModeler(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)
optimizer = optim.SGD(model.parameters(), lr=0.001)
# Freeze embedding layer
#model.freeze_layer('embeddings')
for epoch in range(400):
total_loss = 0
#------- Embedding layers are trained as well here ----#
#lookup_tensor = torch.tensor([word_to_ix["poor"]], dtype=torch.long)
#hello_embed = model.embeddings(lookup_tensor)
#print(hello_embed)
# -----------------------------------------------------#
for context, target in ngrams:
# Step 1. Prepare the inputs to be passed to the model (i.e, turn the words
# into integer indices and wrap them in tensors)
context_idxs = torch.tensor([word_to_ix[w] for w in context], dtype=torch.long)
#print("Context id",context_idxs)
# Step 2. Recall that torch *accumulates* gradients. Before passing in a
# new instance, you need to zero out the gradients from the old
# instance
model.zero_grad()
# Step 3. Run the forward pass, getting log probabilities over next
# words
log_probs = model(context_idxs)
#print(log_probs)
# Step 4. Compute your loss function. (Again, Torch wants the target
# word wrapped in a tensor)
loss = loss_function(log_probs, torch.tensor([word_to_ix[target]], dtype=torch.long))
#print(loss)
# Step 5. Do the backward pass and update the gradient
loss.backward()
optimizer.step()
# Get the Python number from a 1-element Tensor by calling tensor.item()
total_loss += loss.item()
print(total_loss)
losses.append(total_loss)
#print(losses) # The loss decreased every iteration over the training data!
#Print the model layer parameters
#model.print_layer_parameters()
#Predict the next word given n context words
model.predict(['of','all','human'])
model.write_embedding_to_file('embeddings.npy')
cluster_embeddings('embeddings.npy',2)
Skipgrams
The following is a Pytorch implementation of the Skipgram algorithm.
# Author: Srijith Rajamohan
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import urllib.request
from nltk.tokenize import RegexpTokenizer
from nltk.corpus import stopwords
from nltk import word_tokenize
import sklearn
from sklearn.cluster import KMeans
from sklearn.metrics.pairwise import euclidean_distances
torch.manual_seed(1)
CONTEXT_SIZE = 3
EMBEDDING_DIM = 10
def get_key(word_id):
for key,val in word_to_ix.items():
if(val == word_id):
print(key)
def cluster_embeddings(filename,nclusters):
X = np.load(filename)
kmeans = KMeans(n_clusters=nclusters, random_state=0).fit(X)
center = kmeans.cluster_centers_
distances = euclidean_distances(X,center)
for i in np.arange(0,distances.shape[1]):
word_id = np.argmin(distances[:,i])
print(word_id)
get_key(word_id)
def read_data(file_path):
tokenizer = RegexpTokenizer(r'\w+')
data = urllib.request.urlopen(file_path)
data = data.read().decode('utf8')
tokenized_data = word_tokenize(data)
stop_words = set(stopwords.words('english'))
stop_words.update(['.',',',':',';','(',')','#','--','...','"'])
cleaned_words = [ i for i in tokenized_data if i not in stop_words ]
return(cleaned_words)
test_sentence = """Empathy for the poor may not come easily to people who never experienced it. They may blame the victims and insist their predicament can be overcome through determination and hard work.
But they may not realize that extreme poverty can be psychologically and physically incapacitating — a perpetual cycle of bad diets, health care and education exacerbated by the shaming and self-fulfilling prophecies that define it in the public imagination.
Gordon Parks — perhaps more than any artist — saw poverty as “the most savage of all human afflictions” and realized the power of empathy to help us understand it. It was neither an abstract problem nor political symbol, but something he endured growing up destitute in rural Kansas and having spent years documenting poverty throughout the world, including the United States.
That sensitivity informed “Freedom’s Fearful Foe: Poverty,” his celebrated photo essay published in Life magazine in June 1961. He took readers into the lives of a Brazilian boy, Flavio da Silva, and his family, who lived in the ramshackle Catacumba favela in the hills outside Rio de Janeiro. These stark photographs are the subject of a new book, “Gordon Parks: The Flavio Story” (Steidl/The Gordon Parks Foundation), which accompanies a traveling exhibition co-organized by the Ryerson Image Centre in Toronto, where it opens this week, and the J. Paul Getty Museum. Edited with texts by the exhibition’s co-curators, Paul Roth and Amanda Maddox, the book also includes a recent interview with Mr. da Silva and essays by Beatriz Jaguaribe, Maria Alice Rezende de Carvalho and Sérgio Burgi.
""".split()
# we should tokenize the input, but we will ignore that for now
# build a list of tuples. Each tuple is ([ word_i-2, word_i-1 ], target word)
#test_sentence = read_data('https://www.gutenberg.org/files/57884/57884-0.txt')
ngrams = []
for i in range(len(test_sentence) - CONTEXT_SIZE):
tup = [test_sentence[j] for j in np.arange(i + 1 , i + CONTEXT_SIZE + 1) ]
ngrams.append((test_sentence[i],tup))
# print the first 3, just so you can see what they look like
#print(ngrams)
vocab = set(test_sentence)
print("Length of vocabulary",len(vocab))
word_to_ix = {word: i for i, word in enumerate(vocab)}
class SkipgramModeler(nn.Module):
def __init__(self, vocab_size, embedding_dim, context_size):
super(SkipgramModeler, self).__init__()
self.embeddings = nn.Embedding(vocab_size, embedding_dim)
self.linear1 = nn.Linear(embedding_dim, 128)
self.linear2 = nn.Linear(128, context_size * vocab_size)
#self.parameters['context_size'] = context_size
def forward(self, inputs):
embeds = self.embeddings(inputs).view((1, -1)) # -1 implies size inferred for that index from the size of the data
#print(np.mean(np.mean(self.linear2.weight.data.numpy())))
out1 = F.relu(self.linear1(embeds)) # output of first layer
out2 = self.linear2(out1) # output of second layer
#print(embeds)
log_probs = F.log_softmax(out2, dim=1).view(CONTEXT_SIZE,-1)
return log_probs
def predict(self,input):
context_idxs = torch.tensor([word_to_ix[input]], dtype=torch.long)
res = self.forward(context_idxs)
res_arg = torch.argmax(res)
res_val, res_ind = res.sort(descending=True)
indices = [res_ind[i][0] for i in np.arange(0,3)]
for arg in indices:
print( [ (key, val) for key,val in word_to_ix.items() if val == arg ])
def freeze_layer(self,layer):
for name,child in model.named_children():
print(name,child)
if(name == layer):
for names,params in child.named_parameters():
print(names,params)
print(params.size())
params.requires_grad= False
def print_layer_parameters(self):
for name,child in model.named_children():
print(name,child)
for names,params in child.named_parameters():
print(names,params)
print(params.size())
def write_embedding_to_file(self,filename):
for i in self.embeddings.parameters():
weights = i.data.numpy()
np.save(filename,weights)
losses = []
loss_function = nn.NLLLoss()
model = SkipgramModeler(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)
optimizer = optim.SGD(model.parameters(), lr=0.001)
# Freeze embedding layer
#model.freeze_layer('embeddings')
for epoch in range(550):
total_loss = 0
#------- Embedding layers are trained as well here ----#
#lookup_tensor = torch.tensor([word_to_ix["poor"]], dtype=torch.long)
#hello_embed = model.embeddings(lookup_tensor)
#print(hello_embed)
# -----------------------------------------------------#
model.predict('psychologically')
for context, target in ngrams:
# Step 1. Prepare the inputs to be passed to the model (i.e, turn the words
# into integer indices and wrap them in tensors)
#print(context,target)
context_idxs = torch.tensor([word_to_ix[context]], dtype=torch.long)
#print("Context id",context_idxs)
# Step 2. Recall that torch *accumulates* gradients. Before passing in a
# new instance, you need to zero out the gradients from the old
# instance
model.zero_grad()
# Step 3. Run the forward pass, getting log probabilities over next
# words
log_probs = model(context_idxs)
#print(log_probs)
# Step 4. Compute your loss function. (Again, Torch wants the target
# word wrapped in a tensor)
target_list = torch.tensor([word_to_ix[w] for w in target], dtype=torch.long)
loss = loss_function(log_probs, target_list)
#print(loss)
# Step 5. Do the backward pass and update the gradient
loss.backward()
optimizer.step()
# Get the Python number from a 1-element Tensor by calling tensor.item()
total_loss += loss.item()
print(total_loss)
losses.append(total_loss)
#print(losses) # The loss decreased every iteration over the training data!
#Print the model layer parameters
#model.print_layer_parameters()
#Predict the next word given n context words
model.predict('psychologically')
model.write_embedding_to_file('embeddings_skipgrams.npy')
cluster_embeddings('embeddings_skipgrams.npy',5)