With recent advances in machine learning, and, in particular, deep learning methods and optimization algorithms, neural networks become ubiquitous and widely used tools that are used in every aspect of our life. This is why it is important to study and understand their capabilities and limitations. This dissertation studies several particular aspects of learning representation with neural networks. The first aspect involves the question of transferability of learned representation to different domains. In particular, we study different method, models, and algorithms of improving the accuracy of natural language inference task in the medical domain using existing general domain annotated resources, since it is much harder to perform annotation of data in the medical domain. The second aspect we focus on is the problem of decomposition of representation of the input text. In contrast to the widely used representation of the input text as a single fixed-length vector, we present a method of adversarial decomposition the input text into several fixed-length vectors, where each vector captures a specific aspect of the input data, such as "meaning" and "style". The third part presents a problem of bias in machine learning models. At present, it’s especially important as their decisions affect almost every aspect of our life. We present a new way of mitigating bias in application to hiring decisions and classification problems. The proposed approach reduces several prominent biases simultaneously without having access to any protected attributes during the testing or deployment and using only embeddings of people’s names as universal proxies during the training.