Full Code of YBIGTA/DeepNLP-Study for AI

Repository: YBIGTA/DeepNLP-Study
Branch: master
Commit: 5a0f218593c7
Files: 8
Total size: 6.7 KB

Directory structure:
gitextract_i_057m3r/

├── .gitignore
├── CNN_sentence/
│   └── README.md
├── LICENSE
├── QRNN/
│   └── README.md
├── README.md
├── TCML/
│   └── README.md
└── show-attend-and-tell-pytorch/
    ├── README.md
    └── models.py

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# pyenv
.python-version

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# dotenv
.env

# virtualenv
.venv
venv/
ENV/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/

# macOS
.DS_store
.DS_store?


================================================
FILE: CNN_sentence/README.md
================================================
# A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification



## 1. Background

### CNN Architecture

begin with a tokenized sentence -> a sentence matrix

sentence matrix의 row는 word vector들.

word vector들은 pretrained 사용



word vectors dimension: d

length of a given sentence: s

*sentence matrix: s x d*



treat sentence matrix as an ‘image’

perform convolution on it via linear filters

filters with widths equal to the dimensionality of the word vectors (d)

height of the filter: the region size of the filter.



denote the sentence matrix by A

A[i:j]: the sub-matrix of A from row i to row j.



w: weight matrix

b: bias term

The output sequence of the convolution operator: $o_i = w \cdot{A[i:j+h-1]}$

feature map: $c_i = f(o_i + b)$ 	$c \in R^{s-h+1}$

A pooling function is thus applied to each feature map to induce a fixed-length vector.

A common strategy is 1-max pooling

the outputs generated from each filter map can be concatenated into a fixed-length, ‘top-level’ feature vector

then fed through a softmax function to generate the final classification.



## 2. Baseline Models



![Imgur](https://i.imgur.com/s7aXR1N.png)

**CNN architecture for sentence classification**

region sizes -> gram

feature map -> convolution 결과

필터 개수만큼의 feature map을 만들고, Max-pooling 과정을 거쳐 클래스 개수만큼의 스코어를 출력



We kept only the most frequent 30k n-grams for all datasets

tuned hyperparameters via nested cross-fold validation

incorporating word2vec embeddings into feature vectors



### configuration

a linear kernel SVM exploiting uni- and bi-gram features

used averaged word vectors (from Google word2vec or GloVe)



![Imgur](https://i.imgur.com/L7FZiC9.png)



## references

A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional
Neural Networks for Sentence Classification(2016) - Ye Zhang, Byron C. Wallace

http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/

https://ratsgo.github.io/natural%20language%20processing/2017/03/19/CNN/

================================================
FILE: LICENSE
================================================
MIT License

Copyright (c) 2017 YBIGTA

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


================================================
FILE: QRNN/README.md
================================================
https://arxiv.org/abs/1611.01576


================================================
FILE: README.md
================================================
# DeepNLP-Study

딥러닝 기반 자연어처리 스터디의 논문 구현 코드 및 스터디 자료 모음 공간입니다.


## 2018년 7~8월 스터디
- 목표
  - [CMU Neural Networks for NLP 2018](http://phontron.com/class/nn4nlp2017/index.html) 강의를 듣고 발제한다.
  - 매주 모든 사람이 두 강의씩 들으며, 그 중 두 명이 한 강의씩 맡아 발제한다.
  - 발제하지 않는 나머지 사람들은 질문을 하고 발제자는 답변을 한다.
- 강의마다 나온 질문과 답변은 다음 위키 페이지에 모아서 정리한다.
  - [Q&A-for-CMU-Neural-Networks-for-NLP-2018](https://github.com/YBIGTA/DeepNLP-Study/wiki/%5BQ&A-for-CMU-Neural-Networks-for-NLP-2018%5D)
- 참여자 : 김우정, 김지혁, 박성남, 조용래


# 과거 스터디 히스토리

## 2018년 1~2월 스터디 목표
- 목표
  - 이론 - 조재민님의 챗봇을 위한 딥러닝 강의 자료 공부하기 (논문 발제 모음 : [PAPERS REVIEWED](https://github.com/YBIGTA/DeepNLP-Study/wiki/%5BPAPERS-REVIEWED%5D))
    - [Day 01 Introduction to Chatbot](https://www.slideshare.net/JaeminCho6/dl-chatbot-seminar-day-01-80593691)
    - [Day 02 Text Classification with CNN/RNN](https://www.slideshare.net/JaeminCho6/dl-chatbot-seminar-day-02)
    - [Day 03 Conversation Modeling with Seq2Seq / Attention](https://www.slideshare.net/JaeminCho6/dl-chatbot-seminar-day-03)
    - [Day 04 QA with External Memory](https://www.slideshare.net/JaeminCho6/dl-chatbot-seminar-day-04)
  - 구현 - 직접 딥러닝 모델을 한 달에 하나씩 구현해 보기
    - [논문 구현 프로젝트](https://github.com/YBIGTA/DeepNLP-Study/projects/1?)
- [스터디 세부 계획](https://github.com/YBIGTA/DeepNLP-Study/wiki/%5B%EC%8A%A4%ED%84%B0%EB%94%94-%EC%84%B8%EB%B6%80-%EA%B3%84%ED%9A%8D-(2018%EB%85%84-1~2%EC%9B%94)%5D)


## Github 운영 방식

* 모든 스터디 자료는 이 repository를 통해 공유하는 것을 원칙으로 합니다.
* Wiki
  * 스터디 계획, 일정 등의 문서를 작성합니다.
  * 발제를 위한 논문 요약 및 설명 등도 공유합니다.
* Projects
  * 논문 구현 프로젝트 진행 상황을 공유합니다.
  * 스터디 관련 전반적인 할 일들을 관리합니다.
* Code
  * 논문 구현 코드를 논문 별로 폴더를 만들어서 commit합니다.
  * Commit message 대괄호 안에 논문 이름 약어를 적어 commit을 구분합니다.
  * 수정 사항을 영어로 적을 때는 명령문으로, 첫 글자는 대문자로 적습니다.
  * 두 명 이상의 사람이 동시에 같은 파일을 수정하지 않도록 주의합니다.
  * Commit message 예시 : `[논문이름] Do something`



================================================
FILE: TCML/README.md
================================================
# TCML

Meta-Learning with Temporal Convolutions

[https://arxiv.org/abs/1707.03141](https://arxiv.org/abs/1707.03141)



================================================
FILE: show-attend-and-tell-pytorch/README.md
================================================
# Show Attend and Tell

paper: https://arxiv.org/abs/1502.03044

reference: https://github.com/yunjey/show-attend-and-tell

================================================
FILE: show-attend-and-tell-pytorch/models.py
================================================
from torchvision.models import vgg
from torch import nn

## Encoder

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.original_model = vgg.vgg19_bn(pretrained=True)
        self.convs = list(self.original_model.children())[0]
        self.layers = nn.Sequential(*list(self.convs)[:-1])
        
    def forward(self, x):
        x = self.layers(x)
        return x