Abstract: Recently, deep structured networks such as deep convolutional (CNN) and recurrent (RNN) neural networks have enjoyed great success. However, similar to most machine learning techniques, current deep learning approaches rely on conventional statistics and restricted to a specific problem formulation. In particular, they are designed to learn a model for a distribution (or a function) that maps a structured input, typically a vector, matrix, or tensor, to a structured output. Due to this limitation, deep learning has not had any success or significant impact in some applications. In fact, many problems such as object detection, graph problems, and multi-target tracking, are naturally expressed with sets of elements rather than vectors. As opposed to a vector, the size of a set is not fixed in advance, and it is invariant to the ordering of its elements. Therefore, learning approaches built based on conventional statistics cannot be used directly for these problems. In the talk, I will discuss deep learning approaches that use finite set statistics and point processes instead of conventional statistics, as a step towards the set learning problem. I will also highlight further our recent work.