Towards NLP. Страница 5
1054
@towards_nlp
NLP: все n-граммы про анализ текстов. По всем дополнительным вопросам:
-
Towards NLP
Competitive programming with AlphaCode The news that hits me right in the heart: the DeepMind has released AlphaCode — the system that can solve competitive programming tasks. 1. To "solve" competitive programming task we can address it as seq2seq task: given a task description we need to write a code that solves it. 2. So, we can use typical encoder-decoder architecture. 3. And also use the power of pretraining — the model was pretrained on GitHub dataset of code snippets. 4. After that, dataset from competitive programming platform Codeforces was used for fine-tuning. 5. Moreover, this dataset was extended to show model more negative examples. 6. Now, well-prepared model generates very large (millions per sample) candidates of solution. 7. These candidates are filtered to obtain small set of candidates, that will be tested on test cases. The model outperforms 54% of participants of the Codeforces — quite impressive but there is a big room for improvement. Blogpost, paper with model description 
-
Towards NLP
ML and NLP Research Highlights of 2021 by Sebastian Ruder https://ruder.io/ml-highlights-2021/ -
Towards NLP
Adapters in Transformers Have you ever struggle with the problem of too time- and resource-consuming training procedure of Transformers? Especially, when you want not just train the top layers of the model, but indeed train the full model for your task. This is really a big problem. But how it can be avoided? The researchers of the University of Darmstadt have proposed the new paradigm of Transformer-based models training — Adapters. What is this? Let's refer to the picture. Adapter is a one more layer in Transformer block that comes before the final Add&Norm operation. What's the point now — the weights of the original model are frozen, but we train only these adapters! Moreover, once the adapter is pretrained for some task (for instance. for sentiment classification), it can be shared as weights with other users! Now there is special AdapterHub where such Adapters can be shared between all users. All this seems like quite interesting idea! The material is taken from currently going ALPS-2022. [paper] 
-
Реклама
-
Towards NLP
Uncertainty Estimation Continuing my interest in Active Learning, it is impossible to pass by the topic of Uncertainty Estimation of the models. Actually, this is a quite interesting field of ML and super important for applications. For example, in self-driving cars the model should be sure if there is or not a pedestrian on the road. If it is not sure enough if the road is free, it should pass the control to the person. Another very good example: imagine you train a model to classify images of dogs and cats. But suddenly, the model will get an input the image of a bird. It will return you by design the label if it is either cat of dog, but is it sure about its decision? It is important to indicate how your model deals with out-of-domain (OOD) samples, with shifts or new types of data in the test set. Also, uncertainty can appear from the data itself — if it is inconsistencies, if during measurement there were some noises, if it was unfortunate split into train/test/dev and there are shifts in the batches. My list of sources to learn more about Uncertainty Estimation: 1. Youtube lecture from MIT Deep Learning course "Evidential Deep Learning and Uncertainty" [link]. Very good for the general understanding of the problem. 2. A paper (like a book) "A Survey of Uncertainty in Deep Neural Networks" [link] — a comprehensive overview of types of uncertainties in the models, how to measure and how to deal with them. 3. Another book-typed paper "A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges" [link] — also, very good overview now more focused on the techniques how to measure these uncertainties for different model types and tasks. -
Towards NLP
Happy New 2022 Year🎉 I am incredibly thankful to all the readers to be with me this year 🤗! In New Year, I wish you as less as possible of toxic, but more and more positive texts, moments, and relationships in your life! Be active and brave during your learnings and discoveries 💪 Further only higher🚀 -
Towards NLP
Active Learning in NLP My current year is finishing with the start of new research connected with Active Learning and Uncertainty Estimation of different types of NLP models. Currently, I am only sorting out with the topic and literature, but I have already found good resources for the first steps in the field. Here they are: 1. Active Learning Literature Survey: the main book about Active Learning in General. 2. A literature survey of active machine learning in the context of natural language processing: the main survey of Active Learning Techniques specifically for NLP. 3. A Survey of Deep Active Learning: the paper is of 2021, so the modern status of the field is here. 4. How Certain is Your Transformer?: and, of course, I cannot not mention the recent paper from our lab about Uncertainty Estimation of Transformer-based NLP models. If you have any other cool sources about Active Learning topic, you are very welcome to share! I will continue this literature survey in my future posts. Hope that there will be a lot of interesting🕵️♀️ -
Towards NLP
RUSSE-2022 Detoxification Channel If you want to join and participate in RUSSE-2022 Detoxiication task and want to be up to date with all news, join our channel! https://t.me/russe_detox Chat with discussions: https://t.me/joinchat/Ckja7Vh00qPOU887pLonqQ RUSSE-2022 Detoxification task News and announcements about the RUSSE-2022 Detoxification task: https://russe.nlpub.org/2022/tox/ Github repo: https://github.com/skoltech-nlp/russe_detox_2022 Codalab Task: https://codalab.lisn.upsaclay.fr/competitions/642
-
Towards NLP
Yep, this time has come — we are launching our RUSSE-2022 Russian Detoxification task!🎉 What is available now: * Github repo: https://github.com/skoltech-nlp/russe_detox_2022. Here you can find: 1) train and dev parts of the dataset. 2) Delete and t5 baselines with their inference results on dev set. 3) the whole evaluation code that you can run by yourself. * T5 baseline is also available at HuggingFace 🤗: huggingface.co/Skolkov…se-detox * The main platform with leaderboard at Codalab: codalab.lisn.upsaclay.fr/competi…ions/642. To participate, you should register at the platform and make dev set submission. * Submission format: you need to submit .zip ARCHIVE with .txt file where at each row detoxified version of the input is written. You can participate both individually and in team. How to create a team is described in the instruction. * Important dates: - Right now Development phase is open. You can run you experiments on dev set. This phase will be due January, 31. - After that we will release test set without neutral references. To experiment and make submission for test set you will have 2 weeks. Test phase will be closed 14, February. - The best models of the participants will be evaluated manually. The final results of the human evaluation will be published on the website 28, February. -
Towards NLP
RUSSE 2022 Detoxification Competition There was no posts in the channel for a month because we were preparing something quite interesting — the first shared task on text detoxification based on a parallel dataset! The shared task is hold on the base of Dialogue-2022 conference. So, what is going on. The task of text detoxification is quite straightforward: given as an input some toxic text, you need to generate its non-toxic version. For example: Well today i fucking fracking learned something. -> I have learned something new today. Go ahead ban me, i don’t give a shit. -> It won’t matter to me if I get banned. Interesting, right? Previously, I posted here a lot of content about detoxification and our experiments [the first Russian detoxification experiments, SOTA unsupervised English models]. However, all that was mostly about unsupervised methods. We have collected a unique parallel dataset for detoxification with which you are incredibly welcome to experiment! Moreover, your model results will be evaluated manually — we aim to find indeed strong detoxification systems! What is needed from your is to train/find/create such a seq2seq model that will pass human test. This post is a fuse for the track that will start December, 15. More details here: https://russe.nlpub.org/2022/tox/ Telegram group for communication: https://t.me/joinchat/Ckja7Vh00qPOU887pLonqQ See you in two days. -
Towards NLP
OpenAI GPT-3 is now indeed Open Open AI released GPT-3 model which is available via API. Which tasks can you solve? Quite a lot if you write a corresponding prompt — translation, paraphrasing, summarization, question-answering, etc. You can register here and already can start playing with this monster! Of course, at some point there is a need to pay, but for the first start a lot of tokens are available for free. Send you results in the discussions! -
Towards NLP
Text Detoxification using Large Pre-trained Neural Models this whole article is bullshit .This article’s not a good deal. We continue our story about detoxification task. Today at EMNLP conference David Dale (@cointegrated) will present our work about detoxification for English language: paper github. The paper present the usage of two models modificated for text style transfer: - ParaGedi: Gedi model that can perform text generation from scratch guided by a language model informed about specific attributes of a text, e.g. style or topic. We extend this model by enabling it to paraphrase the input text. - CondBERT model: as it is known, BERT model was pretrained on several tasks, one of those is prediction ofmaskedtokens. We can use such task, mask tokens — the attributes of original style — and prediction the substitution for them in our target style. Also, there was trained T5 detoxification model on pseudo-parallel corpus, you can try it via HuggingFace interface 🤗. The proposed models achieve today SOTA in style transfer for detoxification task! You are welcome to test the models and write github issues 🙂 -
Towards NLP
ruDALL-E DALL-E for the Russian language now exists: https://rudalle.ru/ 
-
Towards NLP
Inappropriateness detection In addition to the aforementioned model detecting toxicity in Russian texts, my colleagues collected the dataset of inappropriate utterances and trained the model on it. The 'inappropriateness' is not a substitution of toxicity, it is rather a derivative of toxicity. These are namely the utterances on sensitive topics (the topic which potentially yields unsafe discussion, such as racism, sexism, drugs, LGBT etc) which can, for example, harm the reputation of the company if its dialogue agent replies accordingly. In the same time it is unwanted to block any mention of such sensitive topic by brute-force keywords approach. The published model makes the first step to perform such smart filtering in Russian language. Come and try it!) If you want to learn more about the collected dataset you can learn more in this light-reading article on Skoltech website or in the article published on ACL workshop. P.S. So, there samples in the comments that sometimes current toxicity detection model labels 'toxic' vulnerable topics. We believe, in the future both approaches will be combined in the robust toxicity classifier. -
Towards NLP
RuDetoxification Our work about detoxification for the Russian language was released into habr post (in Russian): https://habr.com/ru/company/ru_mts/blog/585804/ Your are welcome to join the hot discussion 🙃 P.S. The original paper can be found here. -
Towards NLP
Russian Toxicity Classifier In pursuit of the English toxicity classifier, our lab by Nikolay's (@bbkjunior) efforts is also releasing Russian Toxicity Classifier: huggingface.co/Skolkov…assifier Based on BERT, trained on joint datasets from 2ch.hk and ok.ru with F1=0.97. -
Реклама
-
Towards NLP
PhD Skoltech Annual Review Disclaimer for the new readers: I am Daryna, the author of this channel, and I am doing PhD at Skolkovo Institue of Science and Technology which is in Moscow. My PhD is connected with NLP, so that's why I have this channel — to share interesting things that I found out during my research and diving in more deeply in NLP topic. The majority of posts are indeed about NLP, but sometimes I also like to share my personal insights about my going PhD. This posts is of such type. As a PhD student, we have annual evaluation where we share with the committee our progress and are decided to continue our studies or not. For me, it is also time to retrospect the past year in terms of some insights not about research, but about life. There was a similar post last year (but in Russian, sorry🙈). So, I would like to continue the tradition. Here are the second PhD Annual Insights 2021. 1. The main outcome of the outgoing year — science is about collaboration and team work. During my first year, I was working on my topic only by myself (only communicating with my advisor). As a result, the progress of work was changing pretty slowly. The things changed rapidly when I was involved in the project with team of post-doc, researchers and even newly joint Master students. We managed to do a lot of work in less than a year that finished in quite good publications (thanks a lot everybody for that🙏). My point here will be — if you are working alone and struggling with the progress, try to find collaborations with your colleagues in the lab, in other labs, with other research groups. If you look at the most cited/popular papers, they were made by a group of researches. Actually, the whole science is a result of a huge collaborations of people through out time and distance. 2. To write a paper is a huge work. More specifically, the path from an idea and to the final version of the accepted paper with, for example, github repo with code — is a pretty long way of hard work. Last year I wrote that if you have a material and experiments to write about, it is quite easy to write a paper. For submission. But when you are lucky and your paper was accepted, for me, it was like to write one more paper. And then you need to put together and make pretty and workable your code, create a presentation, present the presentation, and discuss it on the conference. That was really like one more project. So, that pieces of paper of about 4-8 pages are the result of tone of work. Be ready and aware of it. 3. Following the previous statement, to be a researcher — to be proactive. Yes, your supervisor can give you tasks and help with direction, but the responsibility of really making work to be done is all on you. If you want your idea to be indeed implemented in live, you are the one who can make it possible. In joint work with your team. 4. To do something even a small piece of work is better than to do nothing at all. It is just a rule of life in general. Just. Remember. This. 5. However, of course, the path to the successful work and a result with submission is fulfilled with obstacles, failures, and disappointments. It can be total chaos wit times when you do not know at all what to do next or even do not have a will to do anything. It is absolutely ok. To make such times easier, I found the way to organize the life outside of my work — make a routine, include sports, eat healthy food, spend time with hobbies. Yes, your work, your research occupies a huge and important part of your life. But you are not only your work. My advice will be to realize yourself outside of your research, so that when some part of your life, work, will be disordered, another part of your life will bring you joy and energy to live. Towards NLP PhD Skoltech Annual Review Пока на канале было небольшое затишье, так как последние дни были в основном посвящены мыслям и подготовке к годовому отчету по аспирантуре. Это также был прекрасный повод поразмыслить, чему же удалось научиться за год, а также, что удалось осознать про науку и свое позиционирование в ней. Ниже я приведу список личных выводов за год, заодно, если будут вопросы, протестируем новую функцию телеграма с комментариями к постам. 1. Аспирантура - это прекрасный опыт. Даже если вы хотите туда пойти чисто ради строчки в резюме. Даже если вы хотите заниматься потом бизнесом и индустриальными проектами (об этом дальше). Даже если вы придете на некоторое время и не закончите. Опыт созерцания и пропускания через себя научного знания всегда будет полезен в жизни. Да, это можно получить и в магистратуре, при особо удачном раскладе, даже в бакалавриате, но аспирантура создает для приобретения научного мышления более концентрированную среду. 2. Писать научные статьи - это не так уж сложно. Просто…
-
Towards NLP
6. Summing up all said above — research is a profession, is a specific unusual style of life. A lot of people outside the academia do not realize that at all. Some people think that you are like an eternal student that cannot do adulthood. I would claim, that research teaches you to be responsible, self-aware, and professional way way better than any industrial work. But this is really just my opinion, different people are comfortable with different styles of life. 7. So, research, again, looks pretty much as an entrepreneurship, isn't it?🙃 8. I will end up with a small part of feminism stuff. I do not like talk too much about such things (I prefer to do😏), but one conclusion hit me and I understood this only almost in the end of the year. Despite the fact that we are living in 2k21, the women in research nowadays are usual situation, and it looks like everybody is with equal rights — there IS A DIFFERENCE between men and women in professional life. Yes, still. Even in so aware and tolerant academia world. And if you go out of academia, this gap in usual life will hit you no less. There are not so many professionals in the world in general, not just good specialists, but real professionals. And among this number, the amount of women is still so small. This is a fact in 2k21. In the end, I wish everybody to aim to the professional level in your work — we can do better, higher and bring impact in the world💪 -
Towards NLP
RoBERTa English Toxicity Classifier We have released our fine-tuned RoBERTa based toxicity classifier for English language on🤗: huggingface.co/Skolkov…assifier The model was trained on the merge of the English parts of the three datasets by Jigsaw. The classifiers perform closely on the test set of the first Jigsaw competition, reaching the AUC-ROC of 0.98 and F1-score of 0.76. So, you can use it now conveniently for any of your research or industrial tasks☺️