<p>I have found many pieces of documentations and recommendations about <code>IEnumerator</code>, <code>IEnumerable</code>, <code>ICollection</code>, <code>IList</code> and their generic counterparts. Sadly, I didn't find yet a tutorial or book which explains the whole hierarchy of interfaces, generic implementations of those interfaces <strong>and</strong> the best usage of each type of those.</p>

<p>What was your best source of knowledge about the generics, their interfaces, and examples of recommended usage of those in C#?</p>

The main way I came to understand them was just by looking at their interfaces.
The inheritance goes like this:
IList : ICollection : IEnumerable
When you look up IEnumerable's interface, notice that it only has the bare minimum necessary to be able to loop through a bunch of items. It doesn't have count or anything else. IEnumerable works well even if what you're looking at is just a stream coming over a wire.
ICollection adds Count. If you are implementing some kind of collection, and if it's easy to know the size of it, then you should implement this. In this case, the generic interface is a little different. ICollection<T> also has methods to modify the collection by adding and removing members. An unordered set would be a good candidate for ICollection.
IList adds the ability to get and set collection members by index. Use this when the order of items is significant. When you have indexes, you can do things like swap the 4th and 5th items to change their order.