It seems that I had it wrong with regard to the impact others can have on you. Having re-read Castaneda on this topic, he contradicts himself but ulitmately tells the truth. He starts by saying:

There is no need for us to say anything about others. There is no need for you or for me to regard other's actions in our thoughts one way or another.

Then later he retorts with:

It is best to erase all personal history because that makes us free from the encumbering thoughts of other people.

How can other peoples thoughts be an encumberment when there is no need to regard them?

The emphasis throughout his work however is on the self being responsible and no one else can effect that, thus he goes on to write:

It is of no use to be sad and complain and feel justified in doing so, believing that someone is always doing something to us. Nobody is doing anything to anybody, much less to a warrior.

I feel that the emphasis placed on the effect of others is that they will effect normal men, but they cannot effect a warrior. People's negative thoughts and actions will bounce off a warrior, so for a man to believe that someone is doing bad things to him is a dissipation of his personal power. He is reflecting his own fears out into the world and making them so. He should really act like a warrior and be inaccessible, therefore cancelling the effect of others thoughts upon him.