I think if your town stole indian lands at founding and gets his borders increased, due to liberty bells, that penalty will also increase, since it would encroach more of indian territory (steal more).
On the other hand, it that town bought land at founding then border increase would not bother indians. You bought the land you take, after all.