In your screen shot I see that the upper range tick mark labels on your x and y axes are partially obscured, or 'clipped' where the meet the plot area boundary. I trust this behaviour is what you wish to change.
You say you call the method setTickMarkClippingModeHigh(), may I please ask what parameter you are passing when you call this method? You are in fact using the correct method. The parameter to be passed in is an Enumeration of type TickMark.ClippingMode. The default value of this Enumeration (thus the default behaviour) is TICKS_AND_LABELS_PERSIST, and it is this behaviour which your screen shot shows.
I trust you wish both tick mark label and line disappear when they reach the plot area boundary, so as to not be displayed partially obscured. As such you would need to call the method on both your x and y axes such as:
Can you please try this and let us know how you get on?
If you are still having problems please post a code snippet and we will do our best to help.